Skip to content

Instantly share code, notes, and snippets.

@Hocdoc

Hocdoc/elasticsearch.log

Created May 20, 2015 08:28
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Hocdoc/0956a3675bd63e0af0d5 to your computer and use it in GitHub Desktop.
Save Hocdoc/0956a3675bd63e0af0d5 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
elasticsearch.log
This file has been truncated, but you can view the full file.
[2015-05-20 00:00:09,192][INFO ][cluster.metadata ] [test1] [.marvel-2015.05.20] creating index, cause [auto(bulk api)], templates [marvel], shards [1]/[1], mappings [_default_, shard_event, index_event, index_stats, node_event, routing_event, cluster_event, cluster_state, cluster_stats, node_stats, indices_stats]
[2015-05-20 00:00:09,267][INFO ][cluster.metadata ] [test1] [.marvel-2015.05.20] update_mapping [node_stats] (dynamic)
[2015-05-20 00:00:09,287][INFO ][cluster.metadata ] [test1] [.marvel-2015.05.20] update_mapping [cluster_event] (dynamic)
[2015-05-20 00:00:09,310][INFO ][cluster.metadata ] [test1] [.marvel-2015.05.20] update_mapping [index_event] (dynamic)
[2015-05-20 00:00:09,311][INFO ][cluster.metadata ] [test1] [.marvel-2015.05.20] update_mapping [routing_event] (dynamic)
[2015-05-20 00:00:09,312][INFO ][cluster.metadata ] [test1] [.marvel-2015.05.20] update_mapping [cluster_state] (dynamic)
[2015-05-20 00:00:09,659][INFO ][cluster.metadata ] [test1] [.marvel-2015.05.20] update_mapping [index_stats] (dynamic)
[2015-05-20 00:00:09,680][INFO ][cluster.metadata ] [test1] [.marvel-2015.05.20] update_mapping [indices_stats] (dynamic)
[2015-05-20 00:00:09,807][INFO ][cluster.metadata ] [test1] [.marvel-2015.05.20] update_mapping [cluster_stats] (dynamic)
[2015-05-20 07:28:24,365][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15529904977 [14.4gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:28:24,484][DEBUG][action.search.type ] [test1] [gardenhose-2015-week15][2], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:28:24,520][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15533579575 [14.4gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:28:24,565][DEBUG][action.search.type ] [test1] [gardenhose-2015-week14][1], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:28:24,605][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15533041917 [14.4gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:28:24,637][DEBUG][action.search.type ] [test1] [gardenhose-2015-week18][0], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:28:24,726][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15550437757 [14.4gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:28:24,727][DEBUG][action.search.type ] [test1] [gardenhose-2015-week20][1], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:28:24,738][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15540102878 [14.4gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:28:24,738][DEBUG][action.search.type ] [test1] [gardenhose-2015-week20][0], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:28:25,704][INFO ][monitor.jvm ] [test1] [gc][young][1114245][71222] duration [716ms], collections [1]/[1.4s], total [716ms]/[2.2h], memory [16.8gb]->[17.2gb]/[23.9gb], all_pools {[young] [76.8mb]->[98.2mb]/[532.5mb]}{[survivor] [66.5mb]->[66.5mb]/[66.5mb]}{[old] [16.7gb]->[17.1gb]/[23.3gb]}
[2015-05-20 07:28:25,723][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15535616907 [14.4gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:28:25,740][DEBUG][action.search.type ] [test1] [gardenhose-2015-week17][0], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:28:25,957][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15532260941 [14.4gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:28:25,957][DEBUG][action.search.type ] [test1] [gardenhose-2015-week15][1], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:28:26,077][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15551252383 [14.4gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:28:26,078][DEBUG][action.search.type ] [test1] [gardenhose-2015-week19][1], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:28:28,022][INFO ][monitor.jvm ] [test1] [gc][young][1114247][71223] duration [918ms], collections [1]/[1.3s], total [918ms]/[2.2h], memory [17.6gb]->[17.5gb]/[23.9gb], all_pools {[young] [510.4mb]->[61.4mb]/[532.5mb]}{[survivor] [66.5mb]->[66.5mb]/[66.5mb]}{[old] [17.1gb]->[17.4gb]/[23.3gb]}
[2015-05-20 07:28:30,537][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15609326797 [14.5gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:28:30,538][DEBUG][action.search.type ] [test1] [gardenhose-2015-week18][2], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:28:30,558][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15614564076 [14.5gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:28:30,558][DEBUG][action.search.type ] [test1] [gardenhose-2015-week17][2], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:28:30,567][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15609425794 [14.5gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:28:30,567][DEBUG][action.search.type ] [test1] [gardenhose-2015-week16][2], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:28:30,653][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15626063385 [14.5gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:28:30,653][DEBUG][action.search.type ] [test1] [gardenhose-2015-week19][2], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:28:30,800][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15629414496 [14.5gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:28:30,808][DEBUG][action.search.type ] [test1] [gardenhose-2015-week20][2], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:28:47,450][WARN ][monitor.jvm ] [test1] [gc][young][1114266][71233] duration [1s], collections [1]/[1.3s], total [1s]/[2.2h], memory [20.1gb]->[20.1gb]/[23.9gb], all_pools {[young] [494.2mb]->[43.5mb]/[532.5mb]}{[survivor] [66.5mb]->[66.5mb]/[66.5mb]}{[old] [19.6gb]->[20gb]/[23.3gb]}
[2015-05-20 07:28:50,823][WARN ][monitor.jvm ] [test1] [gc][young][1114268][71234] duration [1.7s], collections [1]/[2.3s], total [1.7s]/[2.2h], memory [20.4gb]->[20.4gb]/[23.9gb], all_pools {[young] [394.7mb]->[65mb]/[532.5mb]}{[survivor] [66.5mb]->[66.5mb]/[66.5mb]}{[old] [19.9gb]->[20.3gb]/[23.3gb]}
[2015-05-20 07:28:53,452][WARN ][monitor.jvm ] [test1] [gc][young][1114270][71235] duration [1.1s], collections [1]/[1.6s], total [1.1s]/[2.2h], memory [20.6gb]->[20.6gb]/[23.9gb], all_pools {[young] [399.9mb]->[12.1mb]/[532.5mb]}{[survivor] [66.5mb]->[66.5mb]/[66.5mb]}{[old] [20.2gb]->[20.5gb]/[23.3gb]}
[2015-05-20 07:28:57,219][WARN ][monitor.jvm ] [test1] [gc][young][1114272][71236] duration [2.1s], collections [1]/[2.7s], total [2.1s]/[2.2h], memory [20.8gb]->[20.9gb]/[23.9gb], all_pools {[young] [371.8mb]->[32.4mb]/[532.5mb]}{[survivor] [66.5mb]->[66.5mb]/[66.5mb]}{[old] [20.4gb]->[20.8gb]/[23.3gb]}
[2015-05-20 07:29:00,719][WARN ][monitor.jvm ] [test1] [gc][young][1114274][71237] duration [1.8s], collections [1]/[2.5s], total [1.8s]/[2.2h], memory [21.1gb]->[21.2gb]/[23.9gb], all_pools {[young] [337.4mb]->[5.1mb]/[532.5mb]}{[survivor] [66.5mb]->[66.5mb]/[66.5mb]}{[old] [20.7gb]->[21.1gb]/[23.3gb]}
[2015-05-20 07:29:04,300][WARN ][monitor.jvm ] [test1] [gc][young][1114276][71238] duration [1.5s], collections [1]/[2.5s], total [1.5s]/[2.2h], memory [21.3gb]->[21.5gb]/[23.9gb], all_pools {[young] [259.1mb]->[44.2mb]/[532.5mb]}{[survivor] [66.5mb]->[66.5mb]/[66.5mb]}{[old] [21gb]->[21.4gb]/[23.3gb]}
[2015-05-20 07:29:16,625][WARN ][monitor.jvm ] [test1] [gc][young][1114278][71239] duration [1.4s], collections [1]/[11.1s], total [1.4s]/[2.2h], memory [21.9gb]->[17.6gb]/[23.9gb], all_pools {[young] [413.5mb]->[3.1mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [21.4gb]->[17.6gb]/[23.3gb]}
[2015-05-20 07:29:16,625][INFO ][monitor.jvm ] [test1] [gc][old][1114278][164] duration [9.5s], collections [1]/[11.1s], total [9.5s]/[41.4s], memory [21.9gb]->[17.6gb]/[23.9gb], all_pools {[young] [413.5mb]->[3.1mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [21.4gb]->[17.6gb]/[23.3gb]}
[2015-05-20 07:29:49,856][INFO ][monitor.jvm ] [test1] [gc][old][1114290][166] duration [8.2s], collections [1]/[8.6s], total [8.2s]/[49.8s], memory [23.5gb]->[18.5gb]/[23.9gb], all_pools {[young] [506.8mb]->[115.2mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [22.9gb]->[18.4gb]/[23.3gb]}
[2015-05-20 07:30:23,892][INFO ][monitor.jvm ] [test1] [gc][old][1114309][168] duration [8.5s], collections [1]/[8.8s], total [8.5s]/[58.6s], memory [23.5gb]->[19.1gb]/[23.9gb], all_pools {[young] [436.7mb]->[51.1mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23gb]->[19.1gb]/[23.3gb]}
[2015-05-20 07:30:30,421][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15425692161 [14.3gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:30:30,422][DEBUG][action.search.type ] [test1] [gardenhose-2015-week21][2], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:30:55,061][INFO ][monitor.jvm ] [test1] [gc][old][1114330][170] duration [9.4s], collections [1]/[10.6s], total [9.4s]/[1.1m], memory [23.3gb]->[19.7gb]/[23.9gb], all_pools {[young] [105.6mb]->[40.2mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23.2gb]->[19.7gb]/[23.3gb]}
[2015-05-20 07:31:04,825][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15468496746 [14.4gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:31:04,829][DEBUG][action.search.type ] [test1] [gardenhose-2015-week21][1], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:31:09,698][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15473004256 [14.4gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:31:09,699][DEBUG][action.search.type ] [test1] [gardenhose-2015-week21][0], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:31:19,311][INFO ][monitor.jvm ] [test1] [gc][old][1114346][172] duration [7.6s], collections [1]/[7.7s], total [7.6s]/[1.2m], memory [23.3gb]->[20.1gb]/[23.9gb], all_pools {[young] [532.4mb]->[13.9mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [22.8gb]->[20gb]/[23.3gb]}
[2015-05-20 07:31:42,395][INFO ][monitor.jvm ] [test1] [gc][old][1114361][174] duration [7.5s], collections [1]/[8.3s], total [7.5s]/[1.3m], memory [23.3gb]->[20.6gb]/[23.9gb], all_pools {[young] [321.9mb]->[96mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [22.9gb]->[20.5gb]/[23.3gb]}
[2015-05-20 07:32:20,624][INFO ][monitor.jvm ] [test1] [gc][old][1114382][177] duration [8.3s], collections [1]/[8.7s], total [8.3s]/[1.6m], memory [23.6gb]->[21.2gb]/[23.9gb], all_pools {[young] [481.4mb]->[84mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23.1gb]->[21.2gb]/[23.3gb]}
[2015-05-20 07:32:38,802][INFO ][monitor.jvm ] [test1] [gc][old][1114390][178] duration [8.7s], collections [1]/[9.7s], total [8.7s]/[1.8m], memory [23gb]->[21.4gb]/[23.9gb], all_pools {[young] [152.3mb]->[77.4mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [22.8gb]->[21.4gb]/[23.3gb]}
[2015-05-20 07:32:57,141][INFO ][monitor.jvm ] [test1] [gc][old][1114399][179] duration [9.1s], collections [1]/[10s], total [9.1s]/[1.9m], memory [22.9gb]->[21.7gb]/[23.9gb], all_pools {[young] [70.3mb]->[3.9mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [22.7gb]->[21.7gb]/[23.3gb]}
[2015-05-20 07:33:11,945][INFO ][monitor.jvm ] [test1] [gc][old][1114406][180] duration [7.9s], collections [1]/[8.6s], total [7.9s]/[2.1m], memory [23.4gb]->[21.9gb]/[23.9gb], all_pools {[young] [347.8mb]->[75.2mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23gb]->[21.8gb]/[23.3gb]}
[2015-05-20 07:33:26,995][INFO ][monitor.jvm ] [test1] [gc][old][1114414][181] duration [7.5s], collections [1]/[7.8s], total [7.5s]/[2.2m], memory [23.7gb]->[22.1gb]/[23.9gb], all_pools {[young] [498.8mb]->[33.6mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23.1gb]->[22gb]/[23.3gb]}
[2015-05-20 07:33:40,895][INFO ][monitor.jvm ] [test1] [gc][old][1114420][182] duration [8.5s], collections [1]/[8.8s], total [8.5s]/[2.3m], memory [23.4gb]->[22.3gb]/[23.9gb], all_pools {[young] [468.1mb]->[95.4mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [22.9gb]->[22.2gb]/[23.3gb]}
[2015-05-20 07:33:56,354][INFO ][monitor.jvm ] [test1] [gc][old][1114427][183] duration [8.2s], collections [1]/[9.3s], total [8.2s]/[2.5m], memory [23.4gb]->[22.4gb]/[23.9gb], all_pools {[young] [318.5mb]->[31.5mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23gb]->[22.4gb]/[23.3gb]}
[2015-05-20 07:34:10,102][INFO ][monitor.jvm ] [test1] [gc][old][1114433][184] duration [8.4s], collections [1]/[8.5s], total [8.4s]/[2.6m], memory [23.6gb]->[22.7gb]/[23.9gb], all_pools {[young] [501.6mb]->[37mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23.1gb]->[22.6gb]/[23.3gb]}
[2015-05-20 07:34:21,071][INFO ][monitor.jvm ] [test1] [gc][old][1114437][185] duration [7.4s], collections [1]/[7.9s], total [7.4s]/[2.7m], memory [23.3gb]->[22.7gb]/[23.9gb], all_pools {[young] [306.5mb]->[16.2mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23gb]->[22.7gb]/[23.3gb]}
[2015-05-20 07:34:30,246][INFO ][monitor.jvm ] [test1] [gc][old][1114439][186] duration [7.2s], collections [1]/[8.1s], total [7.2s]/[2.9m], memory [23.1gb]->[22.9gb]/[23.9gb], all_pools {[young] [373.4mb]->[10.2mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [22.7gb]->[22.9gb]/[23.3gb]}
[2015-05-20 07:34:37,686][INFO ][monitor.jvm ] [test1] [gc][old][1114441][187] duration [6.2s], collections [1]/[6.4s], total [6.2s]/[3m], memory [23.4gb]->[23gb]/[23.9gb], all_pools {[young] [468.4mb]->[22.9mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [22.9gb]->[22.9gb]/[23.3gb]}
[2015-05-20 07:34:45,520][INFO ][monitor.jvm ] [test1] [gc][old][1114442][188] duration [7.1s], collections [1]/[7.8s], total [7.1s]/[3.1m], memory [23gb]->[23.2gb]/[23.9gb], all_pools {[young] [22.9mb]->[93.7mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [22.9gb]->[23.1gb]/[23.3gb]}
[2015-05-20 07:34:52,622][INFO ][monitor.jvm ] [test1] [gc][old][1114443][189] duration [6.3s], collections [1]/[7.1s], total [6.3s]/[3.2m], memory [23.2gb]->[23.1gb]/[23.9gb], all_pools {[young] [93.7mb]->[12.2mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.1gb]->[23.1gb]/[23.3gb]}
[2015-05-20 07:35:00,554][INFO ][monitor.jvm ] [test1] [gc][old][1114444][190] duration [6.9s], collections [1]/[7.9s], total [6.9s]/[3.3m], memory [23.1gb]->[23.3gb]/[23.9gb], all_pools {[young] [12.2mb]->[96.1mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.1gb]->[23.2gb]/[23.3gb]}
[2015-05-20 07:35:07,718][INFO ][monitor.jvm ] [test1] [gc][old][1114445][191] duration [6.3s], collections [1]/[7.1s], total [6.3s]/[3.4m], memory [23.3gb]->[23.3gb]/[23.9gb], all_pools {[young] [96.1mb]->[26.6mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.2gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:35:16,012][INFO ][monitor.jvm ] [test1] [gc][old][1114447][192] duration [7.1s], collections [1]/[7.2s], total [7.1s]/[3.5m], memory [23.8gb]->[23.4gb]/[23.9gb], all_pools {[young] [494.9mb]->[135.8mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:35:23,105][INFO ][monitor.jvm ] [test1] [gc][old][1114448][193] duration [6.3s], collections [1]/[7s], total [6.3s]/[3.6m], memory [23.4gb]->[23.6gb]/[23.9gb], all_pools {[young] [135.8mb]->[309.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:35:31,642][INFO ][monitor.jvm ] [test1] [gc][old][1114449][194] duration [7.8s], collections [1]/[8.5s], total [7.8s]/[3.8m], memory [23.6gb]->[23.6gb]/[23.9gb], all_pools {[young] [309.3mb]->[256.7mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:35:38,278][INFO ][monitor.jvm ] [test1] [gc][old][1114450][195] duration [6.3s], collections [1]/[6.6s], total [6.3s]/[3.9m], memory [23.6gb]->[23.6gb]/[23.9gb], all_pools {[young] [256.7mb]->[277mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:35:48,465][INFO ][monitor.jvm ] [test1] [gc][old][1114451][196] duration [9.2s], collections [1]/[10.1s], total [9.2s]/[4m], memory [23.6gb]->[23.2gb]/[23.9gb], all_pools {[young] [277mb]->[5.8mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.2gb]/[23.3gb]}
[2015-05-20 07:35:49,355][WARN ][indices.breaker ] [test1] [FIELDDATA] New used memory 15783535783 [14.6gb] from field [message] would be larger than configured breaker: 15420044083 [14.3gb], breaking
[2015-05-20 07:35:49,355][DEBUG][action.search.type ] [test1] [gardenhose-2015-week13][2], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.UncheckedExecutionException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [message] would be larger than limit of [15420044083/14.3gb]
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.circuitBreak(ChildMemoryCircuitBreaker.java:97)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:148)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData$PagedBytesEstimator.beforeLoad(PagedBytesIndexFieldData.java:217)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:89)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 20 more
[2015-05-20 07:35:55,903][INFO ][monitor.jvm ] [test1] [gc][old][1114453][197] duration [6.3s], collections [1]/[6.4s], total [6.3s]/[4.1m], memory [23.7gb]->[23.3gb]/[23.9gb], all_pools {[young] [527.3mb]->[6.7mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.2gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:36:04,055][INFO ][monitor.jvm ] [test1] [gc][old][1114454][198] duration [7.2s], collections [1]/[8.1s], total [7.2s]/[4.2m], memory [23.3gb]->[23.4gb]/[23.9gb], all_pools {[young] [6.7mb]->[99.6mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:36:11,579][INFO ][monitor.jvm ] [test1] [gc][old][1114456][199] duration [6.1s], collections [1]/[6.5s], total [6.1s]/[4.3m], memory [23.7gb]->[23.5gb]/[23.9gb], all_pools {[young] [378.2mb]->[243.9mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:36:19,507][INFO ][monitor.jvm ] [test1] [gc][old][1114457][200] duration [7.3s], collections [1]/[7.9s], total [7.3s]/[4.5m], memory [23.5gb]->[23.8gb]/[23.9gb], all_pools {[young] [243.9mb]->[480.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:36:27,752][INFO ][monitor.jvm ] [test1] [gc][old][1114458][201] duration [7.2s], collections [1]/[8.2s], total [7.2s]/[4.6m], memory [23.8gb]->[23.6gb]/[23.9gb], all_pools {[young] [480.3mb]->[342.5mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:36:35,791][INFO ][monitor.jvm ] [test1] [gc][old][1114459][202] duration [7.2s], collections [1]/[8s], total [7.2s]/[4.7m], memory [23.6gb]->[23.8gb]/[23.9gb], all_pools {[young] [342.5mb]->[514.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:36:42,711][INFO ][monitor.jvm ] [test1] [gc][old][1114460][203] duration [6.6s], collections [1]/[6.9s], total [6.6s]/[4.8m], memory [23.8gb]->[23.7gb]/[23.9gb], all_pools {[young] [514.3mb]->[410.7mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:36:53,097][INFO ][monitor.jvm ] [test1] [gc][old][1114462][204] duration [9.2s], collections [1]/[9.3s], total [9.2s]/[5m], memory [23.8gb]->[23.7gb]/[23.9gb], all_pools {[young] [532.5mb]->[448.1mb]/[532.5mb]}{[survivor] [14.1mb]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:36:58,959][INFO ][monitor.jvm ] [test1] [gc][old][1114463][205] duration [5.5s], collections [1]/[5.8s], total [5.5s]/[5.1m], memory [23.7gb]->[23.9gb]/[23.9gb], all_pools {[young] [448.1mb]->[532.5mb]/[532.5mb]}{[survivor] [0b]->[38mb]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:37:07,774][INFO ][monitor.jvm ] [test1] [gc][old][1114464][206] duration [8.6s], collections [1]/[8.8s], total [8.6s]/[5.2m], memory [23.9gb]->[23.9gb]/[23.9gb], all_pools {[young] [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [38mb]->[40.6mb]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:37:12,885][INFO ][monitor.jvm ] [test1] [gc][old][1114465][207] duration [5s], collections [1]/[5.1s], total [5s]/[5.3m], memory [23.9gb]->[23.9gb]/[23.9gb], all_pools {[young] [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [40.6mb]->[62.2mb]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:38:29,284][DEBUG][action.search.type ] [test1] [gardenhose-2015-week13][0], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.packed.Direct32.<init>(Direct32.java:37)
at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1020)
at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1001)
at org.apache.lucene.util.packed.PackedLongValues$Builder.pack(PackedLongValues.java:266)
at org.apache.lucene.util.packed.PackedLongValues$Builder.pack(PackedLongValues.java:244)
at org.apache.lucene.util.packed.PackedLongValues$Builder.add(PackedLongValues.java:227)
at org.elasticsearch.index.fielddata.ordinals.MultiOrdinals.<init>(MultiOrdinals.java:73)
at org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder.build(OrdinalsBuilder.java:406)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:104)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
[2015-05-20 07:38:29,303][DEBUG][action.search.type ] [test1] [gardenhose-2015-week12][1], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.packed.Direct32.<init>(Direct32.java:37)
at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1020)
at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1001)
at org.apache.lucene.util.packed.PackedLongValues$Builder.pack(PackedLongValues.java:266)
at org.apache.lucene.util.packed.PackedLongValues$Builder.pack(PackedLongValues.java:244)
at org.apache.lucene.util.packed.PackedLongValues$Builder.add(PackedLongValues.java:227)
at org.elasticsearch.index.fielddata.ordinals.MultiOrdinals.<init>(MultiOrdinals.java:73)
at org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder.build(OrdinalsBuilder.java:406)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:104)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
[2015-05-20 07:38:29,297][DEBUG][action.search.type ] [test1] [gardenhose-2015-week15][0], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.packed.Direct32.<init>(Direct32.java:37)
at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1020)
at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1001)
at org.apache.lucene.util.packed.GrowableWriter.<init>(GrowableWriter.java:46)
at org.apache.lucene.util.packed.PagedGrowableWriter.newMutable(PagedGrowableWriter.java:58)
at org.apache.lucene.util.packed.AbstractPagedMutable.resize(AbstractPagedMutable.java:129)
at org.apache.lucene.util.packed.AbstractPagedMutable.grow(AbstractPagedMutable.java:151)
at org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder$OrdinalsStore.newSlice(OrdinalsBuilder.java:175)
at org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder$OrdinalsStore.addOrdinal(OrdinalsBuilder.java:192)
at org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder.addDoc(OrdinalsBuilder.java:330)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:100)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
[2015-05-20 07:38:29,287][INFO ][monitor.jvm ] [test1] [gc][old][1114466][221] duration [1.2m], collections [14]/[1.2m], total [1.2m]/[6.6m], memory [23.9gb]->[23.2gb]/[23.9gb], all_pools {[young] [532.5mb]->[3.9mb]/[532.5mb]}{[survivor] [62.2mb]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.2gb]/[23.3gb]}
[2015-05-20 07:38:29,455][ERROR][marvel.agent.exporter ] [test1] error sending data to [http://0.0.0.0:9200/.marvel-2015.05.20/_bulk]: SocketTimeoutException[Read timed out]
[2015-05-20 07:38:40,935][INFO ][monitor.jvm ] [test1] [gc][old][1114468][222] duration [9.9s], collections [1]/[10.5s], total [9.9s]/[6.7m], memory [23.6gb]->[21.9gb]/[23.9gb], all_pools {[young] [455.6mb]->[36.5mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.2gb]->[21.9gb]/[23.3gb]}
[2015-05-20 07:38:55,775][INFO ][monitor.jvm ] [test1] [gc][old][1114475][223] duration [8s], collections [1]/[8.6s], total [8s]/[6.9m], memory [23.5gb]->[22.3gb]/[23.9gb], all_pools {[young] [321.4mb]->[30.2mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23.1gb]->[22.3gb]/[23.3gb]}
[2015-05-20 07:39:11,868][INFO ][monitor.jvm ] [test1] [gc][old][1114483][224] duration [8.8s], collections [1]/[8.9s], total [8.8s]/[7m], memory [23.6gb]->[22.5gb]/[23.9gb], all_pools {[young] [529mb]->[38.4mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23gb]->[22.5gb]/[23.3gb]}
[2015-05-20 07:39:27,371][INFO ][monitor.jvm ] [test1] [gc][old][1114490][225] duration [8.3s], collections [1]/[9.1s], total [8.3s]/[7.2m], memory [23.7gb]->[22.8gb]/[23.9gb], all_pools {[young] [433.5mb]->[30.4mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23.2gb]->[22.7gb]/[23.3gb]}
[2015-05-20 07:39:42,298][INFO ][monitor.jvm ] [test1] [gc][old][1114495][226] duration [9.8s], collections [1]/[10.5s], total [9.8s]/[7.3m], memory [23.4gb]->[22.6gb]/[23.9gb], all_pools {[young] [297.6mb]->[51.8mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23gb]->[22.6gb]/[23.3gb]}
[2015-05-20 07:39:58,159][WARN ][monitor.jvm ] [test1] [gc][old][1114501][227] duration [10.3s], collections [1]/[10.6s], total [10.3s]/[7.5m], memory [23.7gb]->[22.2gb]/[23.9gb], all_pools {[young] [471mb]->[96.1mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23.2gb]->[22.1gb]/[23.3gb]}
[2015-05-20 07:40:13,807][INFO ][monitor.jvm ] [test1] [gc][old][1114506][228] duration [9.7s], collections [1]/[10.5s], total [9.7s]/[7.7m], memory [23.5gb]->[22.3gb]/[23.9gb], all_pools {[young] [227.9mb]->[66.2mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23.2gb]->[22.3gb]/[23.3gb]}
[2015-05-20 07:40:24,975][INFO ][monitor.jvm ] [test1] [gc][old][1114510][229] duration [7.9s], collections [1]/[8.1s], total [7.9s]/[7.8m], memory [23.6gb]->[22.6gb]/[23.9gb], all_pools {[young] [509.8mb]->[24.6mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23gb]->[22.5gb]/[23.3gb]}
[2015-05-20 07:40:39,760][INFO ][monitor.jvm ] [test1] [gc][old][1114517][230] duration [8s], collections [1]/[8.6s], total [8s]/[7.9m], memory [23.6gb]->[22.9gb]/[23.9gb], all_pools {[young] [445.4mb]->[5.4mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23.1gb]->[22.9gb]/[23.3gb]}
[2015-05-20 07:40:54,727][INFO ][monitor.jvm ] [test1] [gc][old][1114523][231] duration [9.6s], collections [1]/[9.7s], total [9.6s]/[8.1m], memory [23.8gb]->[22.7gb]/[23.9gb], all_pools {[young] [526.3mb]->[35.3mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [23.2gb]->[22.7gb]/[23.3gb]}
[2015-05-20 07:41:07,016][INFO ][monitor.jvm ] [test1] [gc][old][1114528][232] duration [7.4s], collections [1]/[7.8s], total [7.4s]/[8.2m], memory [23.4gb]->[23.1gb]/[23.9gb], all_pools {[young] [430.8mb]->[20.2mb]/[532.5mb]}{[survivor] [66.5mb]->[0b]/[66.5mb]}{[old] [22.9gb]->[23.1gb]/[23.3gb]}
[2015-05-20 07:41:15,347][INFO ][monitor.jvm ] [test1] [gc][old][1114530][233] duration [6.6s], collections [1]/[7.3s], total [6.6s]/[8.3m], memory [23.4gb]->[23.1gb]/[23.9gb], all_pools {[young] [294.5mb]->[19.7mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.1gb]->[23.1gb]/[23.3gb]}
[2015-05-20 07:41:22,378][INFO ][monitor.jvm ] [test1] [gc][old][1114532][234] duration [5.5s], collections [1]/[6s], total [5.5s]/[8.4m], memory [23.5gb]->[23.3gb]/[23.9gb], all_pools {[young] [390.3mb]->[55.7mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.1gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:41:31,597][INFO ][monitor.jvm ] [test1] [gc][old][1114534][235] duration [7s], collections [1]/[8.2s], total [7s]/[8.5m], memory [23.6gb]->[23.5gb]/[23.9gb], all_pools {[young] [340.5mb]->[154.8mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:41:40,887][INFO ][monitor.jvm ] [test1] [gc][old][1114536][236] duration [7.6s], collections [1]/[8.2s], total [7.6s]/[8.7m], memory [23.7gb]->[23.5gb]/[23.9gb], all_pools {[young] [410.4mb]->[230.9mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:41:50,465][INFO ][monitor.jvm ] [test1] [gc][old][1114538][237] duration [7.6s], collections [1]/[8.5s], total [7.6s]/[8.8m], memory [23.8gb]->[23.7gb]/[23.9gb], all_pools {[young] [464.4mb]->[361.8mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:41:59,419][INFO ][monitor.jvm ] [test1] [gc][old][1114539][238] duration [8s], collections [1]/[8.9s], total [8s]/[8.9m], memory [23.7gb]->[23.7gb]/[23.9gb], all_pools {[young] [361.8mb]->[448.9mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:42:07,446][INFO ][monitor.jvm ] [test1] [gc][old][1114540][239] duration [7.2s], collections [1]/[8s], total [7.2s]/[9m], memory [23.7gb]->[23.8gb]/[23.9gb], all_pools {[young] [448.9mb]->[484.4mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:42:15,887][INFO ][monitor.jvm ] [test1] [gc][old][1114541][240] duration [8s], collections [1]/[8.4s], total [8s]/[9.2m], memory [23.8gb]->[23.8gb]/[23.9gb], all_pools {[young] [484.4mb]->[532.5mb]/[532.5mb]}{[survivor] [0b]->[4.6mb]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:42:23,275][INFO ][monitor.jvm ] [test1] [gc][old][1114542][241] duration [7s], collections [1]/[7.3s], total [7s]/[9.3m], memory [23.8gb]->[23.8gb]/[23.9gb], all_pools {[young] [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [4.6mb]->[1.5mb]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:42:32,427][ERROR][marvel.agent.exporter ] [test1] error sending data to [http://0.0.0.0:9200/.marvel-2015.05.20/_bulk]: [SocketTimeoutException[Read timed out]]
[2015-05-20 07:42:32,482][INFO ][monitor.jvm ] [test1] [gc][old][1114543][242] duration [8.9s], collections [1]/[9.2s], total [8.9s]/[9.4m], memory [23.8gb]->[23.8gb]/[23.9gb], all_pools {[young] [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [1.5mb]->[10.5mb]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:43:03,688][INFO ][monitor.jvm ] [test1] [gc][old][1114544][246] duration [30.2s], collections [4]/[31.2s], total [30.2s]/[9.9m], memory [23.8gb]->[23.9gb]/[23.9gb], all_pools {[young] [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [10.5mb]->[52mb]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:44:15,986][DEBUG][action.search.type ] [test1] [gardenhose-2015-week12][0], node[cnieh4FeSrCTwVCA6ocQnQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4bbccc60] lastShard [true]
org.elasticsearch.ElasticsearchException: java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
... 16 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.packed.Direct32.<init>(Direct32.java:37)
at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1020)
at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1001)
at org.apache.lucene.util.packed.GrowableWriter.<init>(GrowableWriter.java:46)
at org.apache.lucene.util.packed.PagedGrowableWriter.newMutable(PagedGrowableWriter.java:58)
at org.apache.lucene.util.packed.AbstractPagedMutable.resize(AbstractPagedMutable.java:129)
at org.apache.lucene.util.packed.AbstractPagedMutable.grow(AbstractPagedMutable.java:151)
at org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder$OrdinalsStore.newSlice(OrdinalsBuilder.java:175)
at org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder$OrdinalsStore.addOrdinal(OrdinalsBuilder.java:208)
at org.elasticsearch.index.fielddata.ordinals.OrdinalsBuilder.addDoc(OrdinalsBuilder.java:330)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:100)
at org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData.loadDirect(PagedBytesIndexFieldData.java:43)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:180)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:167)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:167)
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:74)
at org.elasticsearch.search.aggregations.support.ValuesSource$MetaData.load(ValuesSource.java:88)
at org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:180)
at org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:143)
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.create(ValuesSourceAggregatorFactory.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createAndRegisterContextAware(AggregatorFactories.java:53)
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:157)
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:100)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
[2015-05-20 07:44:15,991][INFO ][monitor.jvm ] [test1] [gc][old][1114545][258] duration [1.1m], collections [12]/[1.2m], total [1.1m]/[11.1m], memory [23.9gb]->[23.7gb]/[23.9gb], all_pools {[young] [532.5mb]->[406.5mb]/[532.5mb]}{[survivor] [52mb]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:44:25,985][INFO ][monitor.jvm ] [test1] [gc][old][1114546][259] duration [9.2s], collections [1]/[9.9s], total [9.2s]/[11.3m], memory [23.7gb]->[23.7gb]/[23.9gb], all_pools {[young] [406.5mb]->[399.2mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:44:35,457][INFO ][node ] [test1] stopping ...
[2015-05-20 07:44:35,518][INFO ][monitor.jvm ] [test1] [gc][old][1114547][260] duration [8.8s], collections [1]/[9.5s], total [8.8s]/[11.4m], memory [23.7gb]->[23.7gb]/[23.9gb], all_pools {[young] [399.2mb]->[412.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [23.3gb]->[23.3gb]/[23.3gb]}
[2015-05-20 07:44:35,817][ERROR][marvel.agent ] [test1] Background thread had an uncaught exception:
org.elasticsearch.ElasticsearchIllegalStateException: Future got interrupted
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:47)
at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:73)
at org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportClusterStats(AgentService.java:314)
at org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:261)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:279)
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:118)
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:45)
... 4 more
[2015-05-20 07:44:35,919][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:577)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:704)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:671)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:348)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:105)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:76)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:67)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:63)
at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:360)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:330)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:323)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performReplicas(TransportShardReplicationOperationAction.java:601)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:516)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:422)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2015-05-20 07:44:45,313][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:577)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:704)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:671)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:348)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:105)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:76)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:67)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:63)
at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:360)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:330)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:323)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performReplicas(TransportShardReplicationOperationAction.java:601)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:516)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:422)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2015-05-20 07:44:45,326][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:577)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:704)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:671)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:348)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:105)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:76)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:67)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:63)
at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:360)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:330)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:323)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performReplicas(TransportShardReplicationOperationAction.java:601)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:516)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:422)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2015-05-20 07:44:45,374][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:577)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:704)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:671)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:348)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:105)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:76)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:67)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:63)
at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:360)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:330)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:323)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performReplicas(TransportShardReplicationOperationAction.java:601)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:516)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:422)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2015-05-20 07:44:45,425][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:577)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:704)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:671)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:348)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:105)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:76)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:67)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:63)
at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:360)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:330)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:323)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performReplicas(TransportShardReplicationOperationAction.java:601)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:516)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:422)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2015-05-20 07:44:45,477][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:577)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:704)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:671)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:348)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:105)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:76)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:67)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:63)
at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:360)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:330)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:323)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performReplicas(TransportShardReplicationOperationAction.java:601)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:516)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:422)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2015-05-20 07:44:45,653][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:577)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:704)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:671)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:348)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:105)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:76)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:67)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:63)
at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:360)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:330)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:323)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performReplicas(TransportShardReplicationOperationAction.java:601)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:516)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:422)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2015-05-20 07:44:45,716][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:577)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:704)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:671)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:348)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:105)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:76)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:67)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:63)
at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:360)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:330)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:323)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performReplicas(TransportShardReplicationOperationAction.java:601)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:516)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:422)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2015-05-20 07:44:45,716][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:577)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:704)
at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:671)
at org.elasticsearch.common.netty.channel.AbstractChannel.write(AbstractChannel.java:348)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:105)
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:76)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:67)
at org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:63)
at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:360)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:330)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:323)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performReplicas(TransportShardReplicationOperationAction.java:601)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:516)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:422)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2015-05-20 07:44:51,496][INFO ][node ] [test1] version[1.5.2], pid[12580], build[62ff986/2015-04-27T09:21:06Z]
[2015-05-20 07:44:51,496][INFO ][node ] [test1] initializing ...
[2015-05-20 07:44:51,567][INFO ][plugins ] [test1] loaded [analysis-icu, marvel], sites [marvel, head, kopf]
[2015-05-20 07:44:58,162][INFO ][node ] [test1] initialized
[2015-05-20 07:44:58,163][INFO ][node ] [test1] starting ...
[2015-05-20 07:44:58,420][INFO ][transport ] [test1] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/10.240.99.146:9300]}
[2015-05-20 07:44:58,435][INFO ][discovery ] [test1] test/GJPnoxLGQAiodkfji18X_A
[2015-05-20 07:45:02,205][INFO ][cluster.service ] [test1] new_master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[/10.240.99.146:9300]], reason: zen-disco-join (elected_as_master)
[2015-05-20 07:45:02,351][INFO ][http ] [test1] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/10.240.99.146:9200]}
[2015-05-20 07:45:02,351][INFO ][node ] [test1] started
[2015-05-20 07:45:03,321][INFO ][gateway ] [test1] recovered [353] indices into cluster_state
[2015-05-20 07:45:06,216][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:45:06,224][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:45:12,541][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:45:12,542][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:45:12,559][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:45:33,932][INFO ][cluster.metadata ] [test1] [.marvel-2015.05.20] update_mapping [index_event] (dynamic)
[2015-05-20 07:45:34,004][INFO ][cluster.metadata ] [test1] [.marvel-2015.05.20] update_mapping [node_event] (dynamic)
[2015-05-20 07:45:34,006][INFO ][cluster.metadata ] [test1] [.marvel-2015.05.20] update_mapping [cluster_event] (dynamic)
[2015-05-20 07:45:53,373][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:45:53,374][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:46:14,430][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:14,431][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:46:14,446][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:14,482][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:14,504][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:20,753][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:20,754][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:46:20,774][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:20,810][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:20,847][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:20,873][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:22,697][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:22,700][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:46:24,178][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:24,179][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:46:24,197][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:24,218][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:24,614][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:24,616][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:46:24,635][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:24,657][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:25,359][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:25,363][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:46:25,665][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:25,666][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:46:25,706][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:25,707][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:46:25,730][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:25,730][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:46:25,823][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:25,825][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:46:25,834][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:25,869][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:25,870][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:46:25,873][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:25,902][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:46:25,908][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:25,910][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:46:32,920][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:32,920][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:46:32,921][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:32,924][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:46:35,264][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:35,265][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:46:35,265][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:35,270][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:46:38,647][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:38,651][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:46:38,652][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:38,653][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:46:42,912][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:42,913][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:46:42,914][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:42,915][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:46:52,911][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:52,912][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:46:52,912][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:46:52,913][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:47:02,911][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:47:02,913][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:47:02,913][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:47:02,914][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:47:12,912][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:47:12,913][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:47:12,913][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:47:12,915][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:47:22,914][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:47:22,915][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:47:22,915][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:47:22,916][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:47:32,911][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:47:32,912][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:47:32,913][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:47:32,915][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:47:42,910][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:47:42,912][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:47:42,912][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:47:42,913][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:47:52,965][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:47:52,967][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:47:52,967][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:47:52,969][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:02,911][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:02,912][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:02,912][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:02,914][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:12,911][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:12,912][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:12,913][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:12,914][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:22,914][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:22,929][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:22,930][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:22,931][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:22,935][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [master [test1][GJPnoxLGQAiodkfji18X_A][backend-test1.c.typeshot-01.internal][inet[backend-test1.c.typeshot-01.internal/10.240.99.146:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-05-20 07:48:22,961][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:22,962][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:22,979][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:22,980][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:22,996][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:22,996][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,013][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,017][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,033][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,034][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,049][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,049][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,065][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,065][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,082][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,082][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,098][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,099][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,115][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,115][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,131][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,131][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,145][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,145][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,158][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,159][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,171][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,172][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,185][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,185][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,198][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,199][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,212][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,212][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,226][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,226][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,240][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,241][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,256][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,256][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,270][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,271][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,285][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,285][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,298][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,299][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,312][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,315][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,329][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,329][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,343][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,343][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,356][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,356][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,369][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,370][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,382][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,383][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,399][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,400][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,411][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,412][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,425][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,426][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,439][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,439][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,454][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,454][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,468][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,468][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,481][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,481][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,494][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,495][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,508][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,509][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,524][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,524][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,538][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,539][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,553][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,553][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,567][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,567][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,580][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,580][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,594][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,595][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,609][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,609][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,623][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,623][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,637][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,637][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,651][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,651][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,664][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,664][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,677][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,678][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,691][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,691][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,705][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,705][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,718][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,719][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,732][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,733][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,746][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,746][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,759][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,759][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,773][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,773][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,787][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,787][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,801][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,801][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,815][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,815][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,829][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,830][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,844][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,844][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,859][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,859][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,874][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,874][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,887][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,888][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,902][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,902][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,916][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,916][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,930][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,930][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,944][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,944][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,958][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,959][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:23,973][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,973][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:23,989][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:23,989][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,002][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,002][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,017][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,017][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,032][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,033][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,047][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,048][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,167][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,168][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,182][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,182][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,196][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,196][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,211][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,211][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,225][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,225][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,239][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,239][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,253][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,253][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,267][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,267][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,280][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,281][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,294][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,294][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,308][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,308][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,321][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,322][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,335][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,335][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,348][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,348][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,360][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,361][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,373][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,374][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,387][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,387][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,400][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,400][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,412][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,412][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,425][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,425][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,437][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,437][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,450][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,450][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,462][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,463][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,475][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,475][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,488][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,488][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,501][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,501][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,514][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,515][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,528][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,528][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,541][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,542][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,555][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,555][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,568][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,568][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,581][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,582][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,595][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,595][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,609][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,609][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,622][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,622][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,636][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,636][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,650][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,651][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,667][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,668][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,679][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,680][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,693][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,693][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,707][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,708][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,722][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,722][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,736][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,736][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,749][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,749][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,762][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,763][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,775][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,775][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,788][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,788][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,801][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,801][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,814][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,814][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,826][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,827][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,839][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,839][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,852][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,852][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,865][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,865][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,877][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,878][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,890][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,891][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,904][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,904][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,917][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,917][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,931][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,931][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:24,945][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,946][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:24,960][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:24,961][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,103][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,103][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,117][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,117][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,132][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,132][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,146][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,146][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,160][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,160][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,175][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,175][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,189][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,189][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,203][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,204][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,218][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,218][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,232][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,232][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,248][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,248][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,264][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,265][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,284][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,285][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,298][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,299][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,313][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,313][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,328][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,328][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,344][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,344][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,363][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,363][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,375][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,376][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,395][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,395][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,415][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,415][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,435][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,435][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,454][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,455][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,475][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,475][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,494][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,494][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,515][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,515][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,535][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,535][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,555][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,556][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,575][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,575][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,593][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,594][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,751][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,751][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,766][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,767][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,780][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,780][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,795][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,795][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,807][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,808][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,823][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,823][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,835][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,836][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,849][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,850][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,863][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,863][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,881][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,881][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,895][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,895][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,909][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,910][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,923][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,924][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,937][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,937][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,950][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,951][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,965][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,965][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:25,981][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,981][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:25,994][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:25,994][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,007][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,007][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,021][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,022][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,035][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,036][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,049][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,050][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,063][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,064][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,078][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,078][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,092][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,092][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,106][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,106][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,121][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,121][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,137][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,138][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,151][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,152][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,166][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,166][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,180][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,180][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,195][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,195][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,209][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,209][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,224][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,224][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,238][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,239][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,253][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,253][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,269][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,269][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,283][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,283][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,296][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,296][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,310][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,310][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,323][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,323][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,337][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,337][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,350][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,351][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,365][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,365][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,379][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,379][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,393][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,393][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,408][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,408][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,422][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,423][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,437][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,437][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,451][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,451][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,465][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,465][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,479][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,479][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,496][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,496][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,515][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,516][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,532][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,532][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,547][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,547][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,561][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,561][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,575][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,576][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week21][1] received shard failed for [gardenhose-2015-week21][1], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [t202D2oMTQaQDxfClEdetg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week21][1] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week21][1] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week21/1/index/segments_32")]; ]]
[2015-05-20 07:48:26,589][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week20][2]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week20][2] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week20][2] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
... 3 more
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119)
... 9 more
[2015-05-20 07:48:26,590][WARN ][cluster.action.shard ] [test1] [gardenhose-2015-week20][2] received shard failed for [gardenhose-2015-week20][2], node[GJPnoxLGQAiodkfji18X_A], [P], s[INITIALIZING], indexUUID [vpihxmWNT12vCpVBwz9FqA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[gardenhose-2015-week20][2] failed recovery]; nested: EngineCreationFailureException[[gardenhose-2015-week20][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/mnt/elasticsearch-test1/test/nodes/0/indices/gardenhose-2015-week20/2/index/segments_c9")]; ]]
[2015-05-20 07:48:26,603][WARN ][indices.cluster ] [test1] [[gardenhose-2015-week21][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [gardenhose-2015-week21][1] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [gardenhose-2015-week21][1] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(Ind
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment