Last active
August 29, 2015 14:00
-
-
Save dblessing/11256763 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We have been struggling with this issue for a few months. We've experienced it in versions 0.90.6 - 0.90.13 and now in | |
1.1, too. | |
We get this error during or after our data loader indexes data. Sometimes it takes a day or two to occur but most recently | |
it's been immediately on/after first index. The shard that fails is always in the same index. This is a 2-node cluster | |
running on CentOS 6.5 with Oracle Java 1.7.0u51. In addition, there is 1 non-data node for the process that handles | |
indexing data and 2 non-data nodes serving the front-end. All non-data nodes are java clients using spring-data-elasticsearch | |
library. All nodes are on 1.1 now. We understand there's a probability that our loader application is causing this but we | |
can't see how or where. Also, it seems like a bug if a client can cause shards to fail on the server. We are grasping at | |
straws now and appreciate any ideas on what could be causing this. Stack below. Thanks for your help. | |
[2014-04-23 15:43:14,141][INFO ][cluster.metadata ] [servicebus-es1.example.com] [product] update_mapping [product] (dynamic) | |
[2014-04-23 15:46:00,040][INFO ][cluster.metadata ] [servicebus-es1.example.com] [product] update_mapping [product] (dynamic) | |
[2014-04-23 15:47:26,884][WARN ][transport ] [servicebus-es1.example.com] Received response for a request that has timed out, sent [31301ms] ago, timed out [1300ms] ago, action [discovery/zen/fd/ping], node [[servicebus-PROD-tomcatapps3.example.com][ewvSIKmYRfS2qKiojcUw7w][tomcatapps3.example.com][inet[/10.0.0.119:9300]]{client=true, data=false, local=false}], id [160544] | |
[2014-04-23 16:30:38,734][INFO ][cluster.service ] [servicebus-es1.example.com] removed {[servicebus-PROD-tomcatapps3.example.com][ewvSIKmYRfS2qKiojcUw7w][tomcatapps3.example.com][inet[/10.0.0.119:9300]]{client=true, data=false, local=false},}, reason: zen-disco-node_left([servicebus-PROD-tomcatapps3.example.com][ewvSIKmYRfS2qKiojcUw7w][tomcatapps3.example.com][inet[/10.0.0.119:9300]]{client=true, data=false, local=false}) | |
[2014-04-23 16:30:49,756][INFO ][cluster.service ] [servicebus-es1.example.com] added {[servicebus-PROD-tomcatapps3.example.com][ixCfue8YRMq3XhOOcdlmEg][tomcatapps3.example.com][inet[/10.0.0.119:9300]]{client=true, data=false, local=false},}, reason: zen-disco-receive(join from node[[servicebus-PROD-tomcatapps3.example.com][ixCfue8YRMq3XhOOcdlmEg][tomcatapps3.example.com][inet[/10.0.0.119:9300]]{client=true, data=false, local=false}]) | |
[2014-04-23 16:35:51,054][WARN ][index.translog ] [servicebus-es1.example.com] [product][4] failed to flush shard on translog threshold | |
org.elasticsearch.index.engine.FlushFailedEngineException: [product][4] Flush failed | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826) | |
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589) | |
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194) | |
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
at java.lang.Thread.run(Thread.java:744) | |
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed | |
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56) | |
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81) | |
at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45) | |
at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142) | |
at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526) | |
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136) | |
at org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284) | |
at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164) | |
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155) | |
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874) | |
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) | |
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989) | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812) | |
... 5 more | |
[2014-04-23 16:35:54,599][WARN ][index.translog ] [servicebus-es1.example.com] [product][1] failed to flush shard on translog threshold | |
org.elasticsearch.index.engine.FlushFailedEngineException: [product][1] Flush failed | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826) | |
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589) | |
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194) | |
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
at java.lang.Thread.run(Thread.java:744) | |
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed | |
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56) | |
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81) | |
at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45) | |
at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142) | |
at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526) | |
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136) | |
at org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284) | |
at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164) | |
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155) | |
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874) | |
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) | |
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989) | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812) | |
... 5 more | |
[2014-04-23 16:36:02,389][WARN ][index.translog ] [servicebus-es1.example.com] [product][1] failed to flush shard on translog threshold | |
org.elasticsearch.index.engine.FlushFailedEngineException: [product][1] Flush failed | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826) | |
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589) | |
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194) | |
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
at java.lang.Thread.run(Thread.java:744) | |
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed | |
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56) | |
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81) | |
at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45) | |
at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142) | |
at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526) | |
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136) | |
at org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284) | |
at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164) | |
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155) | |
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874) | |
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) | |
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989) | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812) | |
... 5 more | |
[2014-04-23 16:36:06,758][WARN ][index.translog ] [servicebus-es1.example.com] [product][4] failed to flush shard on translog threshold | |
org.elasticsearch.index.engine.FlushFailedEngineException: [product][4] Flush failed | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826) | |
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589) | |
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194) | |
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
at java.lang.Thread.run(Thread.java:744) | |
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed | |
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56) | |
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81) | |
at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45) | |
at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142) | |
at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526) | |
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136) | |
at org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284) | |
at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164) | |
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155) | |
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874) | |
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) | |
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989) | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812) | |
... 5 more | |
[2014-04-23 16:36:11,683][WARN ][index.translog ] [servicebus-es1.example.com] [product][1] failed to flush shard on translog threshold | |
org.elasticsearch.index.engine.FlushFailedEngineException: [product][1] Flush failed | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826) | |
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589) | |
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194) | |
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
at java.lang.Thread.run(Thread.java:744) | |
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed | |
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56) | |
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81) | |
at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45) | |
at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142) | |
at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526) | |
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136) | |
at org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284) | |
at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164) | |
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155) | |
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874) | |
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) | |
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989) | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812) | |
... 5 more | |
[2014-04-23 16:36:14,900][WARN ][index.translog ] [servicebus-es1.example.com] [product][4] failed to flush shard on translog threshold | |
org.elasticsearch.index.engine.FlushFailedEngineException: [product][4] Flush failed | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826) | |
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589) | |
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194) | |
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
at java.lang.Thread.run(Thread.java:744) | |
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed | |
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56) | |
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81) | |
at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45) | |
at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142) | |
at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526) | |
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136) | |
at org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284) | |
at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164) | |
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155) | |
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874) | |
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) | |
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989) | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812) | |
... 5 more | |
[2014-04-23 16:36:23,382][WARN ][index.translog ] [servicebus-es1.example.com] [product][4] failed to flush shard on translog threshold | |
org.elasticsearch.index.engine.FlushFailedEngineException: [product][4] Flush failed | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826) | |
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589) | |
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194) | |
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
at java.lang.Thread.run(Thread.java:744) | |
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed | |
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56) | |
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81) | |
at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45) | |
at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142) | |
at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526) | |
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136) | |
at org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284) | |
at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164) | |
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155) | |
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874) | |
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) | |
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989) | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812) | |
... 5 more | |
[2014-04-23 16:36:24,811][WARN ][index.translog ] [servicebus-es1.example.com] [product][1] failed to flush shard on translog threshold | |
org.elasticsearch.index.engine.FlushFailedEngineException: [product][1] Flush failed | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826) | |
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589) | |
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194) | |
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
at java.lang.Thread.run(Thread.java:744) | |
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed | |
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56) | |
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81) | |
at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45) | |
at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142) | |
at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526) | |
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136) | |
at org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284) | |
at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164) | |
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155) | |
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874) | |
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) | |
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989) | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812) | |
... 5 more | |
[2014-04-23 16:36:28,577][WARN ][index.translog ] [servicebus-es1.example.com] [product][4] failed to flush shard on translog threshold | |
org.elasticsearch.index.engine.FlushFailedEngineException: [product][4] Flush failed | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826) | |
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589) | |
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194) | |
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
at java.lang.Thread.run(Thread.java:744) | |
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed | |
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56) | |
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81) | |
at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45) | |
at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142) | |
at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526) | |
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136) | |
at org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284) | |
at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164) | |
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155) | |
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874) | |
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) | |
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989) | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812) | |
... 5 more | |
[2014-04-23 16:36:41,739][WARN ][index.translog ] [servicebus-es1.example.com] [product][1] failed to flush shard on translog threshold | |
org.elasticsearch.index.engine.FlushFailedEngineException: [product][1] Flush failed | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:826) | |
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589) | |
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194) | |
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
at java.lang.Thread.run(Thread.java:744) | |
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed | |
at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:253) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:102) | |
at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:56) | |
at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:503) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.contains(DeleteByQueryWrappingFilter.java:122) | |
at org.elasticsearch.index.search.child.DeleteByQueryWrappingFilter.getDocIdSet(DeleteByQueryWrappingFilter.java:81) | |
at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45) | |
at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:142) | |
at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:526) | |
at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136) | |
at org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:546) | |
at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284) | |
at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3164) | |
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3155) | |
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874) | |
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) | |
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989) | |
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812) | |
... 5 more | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment