Skip to content

Instantly share code, notes, and snippets.

@bittusarkar

bittusarkar/es-d01-rm.log

Created December 16, 2018 06:36
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bittusarkar/8b4adb4573642508324115aec86ff211 to your computer and use it in GitHub Desktop.
Save bittusarkar/8b4adb4573642508324115aec86ff211 to your computer and use it in GitHub Desktop.
Download ZIP
Shard recovery logs
Raw
es-d01-rm.log
[2018-12-15T06:14:12,469][INFO ][o.e.c.s.ClusterSettings ] [es-d01-rm] updating [cluster.routing.allocation.cluster_concurrent_rebalance] from [2] to [10]
[2018-12-15T06:36:33,680][DEBUG][o.e.a.b.TransportShardBulkAction] [workitemsearchshared_8_3][10] failed to execute bulk item (update) BulkShardRequest [[workitemsearchshared_8_3][10]] containing [29] requests
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [fields.real]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:302) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:607) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseNonDynamicArray(DocumentParser.java:594) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseArray(DocumentParser.java:542) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:396) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:482) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:500) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:394) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:67) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:261) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:714) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:692) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:673) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:548) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeUpdateRequestOnce(TransportShardBulkAction.java:359) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeUpdateRequest(TransportShardBulkAction.java:406) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:240) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:123) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:110) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:72) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1034) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1012) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:103) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:359) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:299) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:975) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:972) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2221) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:984) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:98) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:320) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:295) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:282) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:656) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.IllegalArgumentException: [float] supports only finite values, but got [Infinity]
at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$2.validateParsed(NumberFieldMapper.java:351) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$2.parse(NumberFieldMapper.java:285) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$2.parse(NumberFieldMapper.java:265) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:996) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:297) ~[elasticsearch-6.2.4.jar:6.2.4]
... 45 more
[2018-12-15T06:47:34,678][DEBUG][o.e.a.b.TransportShardBulkAction] [workitemsearchshared_8_3][10] failed to execute bulk item (update) BulkShardRequest [[workitemsearchshared_8_3][10]] containing [29] requests
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [fields.real]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:302) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:607) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseNonDynamicArray(DocumentParser.java:594) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseArray(DocumentParser.java:542) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:396) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:482) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:500) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:394) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:67) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:261) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:714) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:692) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:673) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:548) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeUpdateRequestOnce(TransportShardBulkAction.java:359) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeUpdateRequest(TransportShardBulkAction.java:406) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:240) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:123) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:110) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:72) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1034) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1012) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:103) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:359) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:299) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:975) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:972) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2221) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:984) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:98) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:320) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:295) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:282) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:656) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.IllegalArgumentException: [float] supports only finite values, but got [Infinity]
at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$2.validateParsed(NumberFieldMapper.java:351) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$2.parse(NumberFieldMapper.java:285) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$2.parse(NumberFieldMapper.java:265) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:996) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:297) ~[elasticsearch-6.2.4.jar:6.2.4]
... 45 more
[2018-12-15T06:58:35,478][DEBUG][o.e.a.b.TransportShardBulkAction] [workitemsearchshared_8_3][10] failed to execute bulk item (update) BulkShardRequest [[workitemsearchshared_8_3][10]] containing [29] requests
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [fields.real]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:302) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:607) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseNonDynamicArray(DocumentParser.java:594) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseArray(DocumentParser.java:542) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:396) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:482) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:500) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:394) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:67) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:261) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:714) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:692) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:673) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:548) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeUpdateRequestOnce(TransportShardBulkAction.java:359) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeUpdateRequest(TransportShardBulkAction.java:406) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:240) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:123) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:110) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:72) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1034) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1012) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:103) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:359) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:299) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:975) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:972) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2221) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:984) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:98) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:320) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:295) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:282) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:656) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.IllegalArgumentException: [float] supports only finite values, but got [Infinity]
at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$2.validateParsed(NumberFieldMapper.java:351) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$2.parse(NumberFieldMapper.java:285) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$2.parse(NumberFieldMapper.java:265) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:996) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:297) ~[elasticsearch-6.2.4.jar:6.2.4]
... 45 more
Raw
es-d08-rm.log
[2018-12-15T02:23:00,634][WARN ][o.e.i.c.IndicesClusterStateService] [es-d08-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T03:23:01,670][WARN ][o.e.i.c.IndicesClusterStateService] [es-d08-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T06:14:12,670][INFO ][o.e.c.s.ClusterSettings ] [es-d08-rm] updating [cluster.routing.allocation.cluster_concurrent_rebalance] from [2] to [10]
[2018-12-15T06:23:04,856][WARN ][o.e.i.c.IndicesClusterStateService] [es-d08-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T07:23:05,878][WARN ][o.e.i.c.IndicesClusterStateService] [es-d08-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T10:23:09,108][WARN ][o.e.i.c.IndicesClusterStateService] [es-d08-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T11:23:10,141][WARN ][o.e.i.c.IndicesClusterStateService] [es-d08-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T13:23:13,409][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] started recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [392]
[2018-12-15T13:23:13,409][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] collecting local files for [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T13:23:13,409][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] [392] shard folder empty, recovering all files
[2018-12-15T13:23:13,409][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] local file count [0]
[2018-12-15T13:23:13,409][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] preparing for file-based recovery from [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T13:23:13,409][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] preparing shard for peer recovery
[2018-12-15T13:23:13,409][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] starting recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}
[2018-12-15T13:53:13,427][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [monitor] rescheduling check for [392]. last access time is [270705215173500]
[2018-12-15T14:23:13,454][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] failing recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [392]. Send shard failure: [true]
[2018-12-15T14:23:13,454][WARN ][o.e.i.c.IndicesClusterStateService] [es-d08-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T14:23:13,454][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] recovery cancelled
org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: operation was cancelled reason [failed recovery [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]];
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchTimeoutException[no activity after [30m]]
... 6 more
]]
at org.elasticsearch.common.util.CancellableThreads.onCancel(CancellableThreads.java:63) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:129) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Suppressed: java.lang.IllegalStateException: Future got interrupted
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:32) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:202) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_72]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_72]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44) ~[elasticsearch-6.2.4.jar:6.2.4]
... 12 more
[2018-12-15T14:23:14,532][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] started recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [393]
[2018-12-15T14:23:14,532][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] collecting local files for [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T14:23:14,532][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] [393] shard folder empty, recovering all files
[2018-12-15T14:23:14,532][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] local file count [0]
[2018-12-15T14:23:14,532][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] preparing for file-based recovery from [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T14:23:14,532][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] preparing shard for peer recovery
[2018-12-15T14:23:14,532][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] starting recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}
[2018-12-15T14:53:14,558][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [monitor] rescheduling check for [393]. last access time is [274306317320299]
[2018-12-15T15:23:14,584][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] failing recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [393]. Send shard failure: [true]
[2018-12-15T15:23:14,584][WARN ][o.e.i.c.IndicesClusterStateService] [es-d08-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T15:23:14,584][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] recovery cancelled
org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: operation was cancelled reason [failed recovery [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]];
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchTimeoutException[no activity after [30m]]
... 6 more
]]
at org.elasticsearch.common.util.CancellableThreads.onCancel(CancellableThreads.java:63) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:129) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Suppressed: java.lang.IllegalStateException: Future got interrupted
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:32) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:202) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_72]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_72]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44) ~[elasticsearch-6.2.4.jar:6.2.4]
... 12 more
[2018-12-15T17:23:17,871][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] started recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [394]
[2018-12-15T17:23:17,871][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] collecting local files for [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T17:23:17,871][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] [394] shard folder empty, recovering all files
[2018-12-15T17:23:17,871][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] local file count [0]
[2018-12-15T17:23:17,871][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] preparing for file-based recovery from [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T17:23:17,871][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] preparing shard for peer recovery
[2018-12-15T17:23:17,871][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] starting recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}
[2018-12-15T17:53:17,889][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [monitor] rescheduling check for [394]. last access time is [285109586203900]
[2018-12-15T18:23:17,914][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] failing recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [394]. Send shard failure: [true]
[2018-12-15T18:23:17,914][WARN ][o.e.i.c.IndicesClusterStateService] [es-d08-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T18:23:17,914][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] recovery cancelled
org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: operation was cancelled reason [failed recovery [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]];
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchTimeoutException[no activity after [30m]]
... 6 more
]]
at org.elasticsearch.common.util.CancellableThreads.onCancel(CancellableThreads.java:63) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:129) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Suppressed: java.lang.IllegalStateException: Future got interrupted
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:32) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:202) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_72]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_72]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44) ~[elasticsearch-6.2.4.jar:6.2.4]
... 12 more
[2018-12-15T18:23:18,835][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] started recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [395]
[2018-12-15T18:23:18,835][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] collecting local files for [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T18:23:18,835][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] [395] shard folder empty, recovering all files
[2018-12-15T18:23:18,835][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] local file count [0]
[2018-12-15T18:23:18,835][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] preparing for file-based recovery from [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T18:23:18,835][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] preparing shard for peer recovery
[2018-12-15T18:23:18,835][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] starting recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}
[2018-12-15T18:53:18,866][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [monitor] rescheduling check for [395]. last access time is [288710531394100]
[2018-12-15T19:23:18,882][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] failing recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [395]. Send shard failure: [true]
[2018-12-15T19:23:18,882][WARN ][o.e.i.c.IndicesClusterStateService] [es-d08-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T19:23:18,882][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] recovery cancelled
org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: operation was cancelled reason [failed recovery [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]];
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchTimeoutException[no activity after [30m]]
... 6 more
]]
at org.elasticsearch.common.util.CancellableThreads.onCancel(CancellableThreads.java:63) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:129) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Suppressed: java.lang.IllegalStateException: Future got interrupted
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:32) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:202) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_72]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_72]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44) ~[elasticsearch-6.2.4.jar:6.2.4]
... 12 more
[2018-12-15T21:23:22,222][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] started recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [396]
[2018-12-15T21:23:22,222][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] collecting local files for [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T21:23:22,222][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] [396] shard folder empty, recovering all files
[2018-12-15T21:23:22,222][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] local file count [0]
[2018-12-15T21:23:22,222][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] preparing for file-based recovery from [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T21:23:22,222][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] preparing shard for peer recovery
[2018-12-15T21:23:22,222][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] starting recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}
[2018-12-15T21:53:22,239][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [monitor] rescheduling check for [396]. last access time is [299513840928300]
[2018-12-15T22:23:22,265][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] failing recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [396]. Send shard failure: [true]
[2018-12-15T22:23:22,265][WARN ][o.e.i.c.IndicesClusterStateService] [es-d08-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T22:23:22,265][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] recovery cancelled
org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: operation was cancelled reason [failed recovery [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]];
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchTimeoutException[no activity after [30m]]
... 6 more
]]
at org.elasticsearch.common.util.CancellableThreads.onCancel(CancellableThreads.java:63) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:129) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Suppressed: java.lang.IllegalStateException: Future got interrupted
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:32) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:202) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_72]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_72]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44) ~[elasticsearch-6.2.4.jar:6.2.4]
... 12 more
[2018-12-15T22:23:23,186][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] started recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [397]
[2018-12-15T22:23:23,186][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] collecting local files for [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T22:23:23,186][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] [397] shard folder empty, recovering all files
[2018-12-15T22:23:23,186][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] local file count [0]
[2018-12-15T22:23:23,186][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] preparing for file-based recovery from [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T22:23:23,186][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] preparing shard for peer recovery
[2018-12-15T22:23:23,186][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] starting recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}
[2018-12-15T22:53:23,213][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [monitor] rescheduling check for [397]. last access time is [303114788915800]
[2018-12-15T23:23:23,228][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] failing recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [397]. Send shard failure: [true]
[2018-12-15T23:23:23,228][WARN ][o.e.i.c.IndicesClusterStateService] [es-d08-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T23:23:23,228][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d08-rm] recovery cancelled
org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: operation was cancelled reason [failed recovery [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]];
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchTimeoutException[no activity after [30m]]
... 6 more
]]
at org.elasticsearch.common.util.CancellableThreads.onCancel(CancellableThreads.java:63) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:129) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Suppressed: java.lang.IllegalStateException: Future got interrupted
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:32) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:202) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_72]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_72]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44) ~[elasticsearch-6.2.4.jar:6.2.4]
... 12 more
Raw
es-d10-rm.log
[2018-12-15T00:22:58,405][WARN ][o.e.i.c.IndicesClusterStateService] [es-d10-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T01:22:59,509][WARN ][o.e.i.c.IndicesClusterStateService] [es-d10-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T04:23:02,754][WARN ][o.e.i.c.IndicesClusterStateService] [es-d10-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T05:23:03,889][WARN ][o.e.i.c.IndicesClusterStateService] [es-d10-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T06:14:12,659][INFO ][o.e.c.s.ClusterSettings ] [es-d10-rm] updating [cluster.routing.allocation.cluster_concurrent_rebalance] from [2] to [10]
[2018-12-15T08:23:06,924][WARN ][o.e.i.c.IndicesClusterStateService] [es-d10-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T09:23:08,041][WARN ][o.e.i.c.IndicesClusterStateService] [es-d10-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T12:23:11,228][WARN ][o.e.i.c.IndicesClusterStateService] [es-d10-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T13:23:12,312][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] failing recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [441]. Send shard failure: [true]
[2018-12-15T13:23:12,312][WARN ][o.e.i.c.IndicesClusterStateService] [es-d10-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T13:23:12,312][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] recovery cancelled
org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: operation was cancelled reason [failed recovery [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]];
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchTimeoutException[no activity after [30m]]
... 6 more
]]
at org.elasticsearch.common.util.CancellableThreads.onCancel(CancellableThreads.java:63) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:129) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Suppressed: java.lang.IllegalStateException: Future got interrupted
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:32) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:202) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_72]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_72]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44) ~[elasticsearch-6.2.4.jar:6.2.4]
... 12 more
[2018-12-15T15:23:15,599][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] started recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [442]
[2018-12-15T15:23:15,599][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] collecting local files for [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T15:23:15,599][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] [442] shard folder empty, recovering all files
[2018-12-15T15:23:15,599][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] local file count [0]
[2018-12-15T15:23:15,599][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] preparing for file-based recovery from [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T15:23:15,599][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] preparing shard for peer recovery
[2018-12-15T15:23:15,599][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] starting recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}
[2018-12-15T15:53:15,630][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [monitor] rescheduling check for [442]. last access time is [274143261910200]
[2018-12-15T16:23:15,660][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] failing recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [442]. Send shard failure: [true]
[2018-12-15T16:23:15,660][WARN ][o.e.i.c.IndicesClusterStateService] [es-d10-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T16:23:15,660][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] recovery cancelled
org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: operation was cancelled reason [failed recovery [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]];
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchTimeoutException[no activity after [30m]]
... 6 more
]]
at org.elasticsearch.common.util.CancellableThreads.onCancel(CancellableThreads.java:63) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:129) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Suppressed: java.lang.IllegalStateException: Future got interrupted
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:32) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:202) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_72]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_72]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44) ~[elasticsearch-6.2.4.jar:6.2.4]
... 12 more
[2018-12-15T16:23:16,739][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] started recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [443]
[2018-12-15T16:23:16,739][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] collecting local files for [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T16:23:16,739][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] [443] shard folder empty, recovering all files
[2018-12-15T16:23:16,739][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] local file count [0]
[2018-12-15T16:23:16,739][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] preparing for file-based recovery from [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T16:23:16,739][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] preparing shard for peer recovery
[2018-12-15T16:23:16,739][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] starting recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}
[2018-12-15T16:53:16,768][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [monitor] rescheduling check for [443]. last access time is [277744358268000]
[2018-12-15T17:23:16,790][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] failing recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [443]. Send shard failure: [true]
[2018-12-15T17:23:16,790][WARN ][o.e.i.c.IndicesClusterStateService] [es-d10-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T17:23:16,790][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] recovery cancelled
org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: operation was cancelled reason [failed recovery [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]];
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchTimeoutException[no activity after [30m]]
... 6 more
]]
at org.elasticsearch.common.util.CancellableThreads.onCancel(CancellableThreads.java:63) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:129) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Suppressed: java.lang.IllegalStateException: Future got interrupted
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:32) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:202) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_72]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_72]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44) ~[elasticsearch-6.2.4.jar:6.2.4]
... 12 more
[2018-12-15T19:23:19,957][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] started recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [444]
[2018-12-15T19:23:19,957][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] collecting local files for [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T19:23:19,957][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] [444] shard folder empty, recovering all files
[2018-12-15T19:23:19,973][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] local file count [0]
[2018-12-15T19:23:19,973][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] preparing for file-based recovery from [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T19:23:19,973][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] preparing shard for peer recovery
[2018-12-15T19:23:19,973][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] starting recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}
[2018-12-15T19:53:19,999][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [monitor] rescheduling check for [444]. last access time is [288547479005700]
[2018-12-15T20:23:20,027][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] failing recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [444]. Send shard failure: [true]
[2018-12-15T20:23:20,027][WARN ][o.e.i.c.IndicesClusterStateService] [es-d10-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T20:23:20,027][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] recovery cancelled
org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: operation was cancelled reason [failed recovery [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]];
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchTimeoutException[no activity after [30m]]
... 6 more
]]
at org.elasticsearch.common.util.CancellableThreads.onCancel(CancellableThreads.java:63) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:129) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Suppressed: java.lang.IllegalStateException: Future got interrupted
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:32) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:202) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_72]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_72]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44) ~[elasticsearch-6.2.4.jar:6.2.4]
... 12 more
[2018-12-15T20:23:21,068][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] started recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [445]
[2018-12-15T20:23:21,068][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] collecting local files for [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T20:23:21,068][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] [445] shard folder empty, recovering all files
[2018-12-15T20:23:21,068][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] local file count [0]
[2018-12-15T20:23:21,068][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] preparing for file-based recovery from [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T20:23:21,068][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] preparing shard for peer recovery
[2018-12-15T20:23:21,068][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] starting recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}
[2018-12-15T20:53:21,100][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [monitor] rescheduling check for [445]. last access time is [292148550841800]
[2018-12-15T21:23:21,126][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] failing recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [445]. Send shard failure: [true]
[2018-12-15T21:23:21,126][WARN ][o.e.i.c.IndicesClusterStateService] [es-d10-rm] [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T21:23:21,126][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] recovery cancelled
org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: operation was cancelled reason [failed recovery [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]];
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ElasticsearchTimeoutException[no activity after [30m]]
... 6 more
]]
at org.elasticsearch.common.util.CancellableThreads.onCancel(CancellableThreads.java:63) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:129) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Suppressed: java.lang.IllegalStateException: Future got interrupted
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:32) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.lambda$doRecovery$1(PeerRecoveryTargetService.java:202) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:195) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:81) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:635) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_72]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_72]
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44) ~[elasticsearch-6.2.4.jar:6.2.4]
... 12 more
[2018-12-15T23:23:24,305][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] started recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}, id [446]
[2018-12-15T23:23:24,305][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] collecting local files for [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T23:23:24,305][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] [446] shard folder empty, recovering all files
[2018-12-15T23:23:24,305][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] local file count [0]
[2018-12-15T23:23:24,305][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] preparing for file-based recovery from [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}]
[2018-12-15T23:23:24,305][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] preparing shard for peer recovery
[2018-12-15T23:23:24,305][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] starting recovery from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}
[2018-12-15T23:53:24,335][TRACE][o.e.i.r.PeerRecoveryTargetService] [es-d10-rm] [monitor] rescheduling check for [446]. last access time is [302951669888800]
Raw
es-m02-rm.log
[2018-12-15T00:22:58,276][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]], allocation id [UihGynYUTaiIggwjnM5JfA], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T00:48:32,848][INFO ][o.e.c.m.MetaDataMappingService] [es-m02-rm] [workitem_workitemcontract_1421_shared_be55fbba-11ce-4afb-8105-4eb7c98f4323/l8U0ZzP0R12NleTJ02F7tA] update_mapping [workItemContract]
[2018-12-15T01:22:59,393][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]], allocation id [F1ykaHOcQ5aHvYTXV_KYug], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T02:23:00,497][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]], allocation id [1JS4pN4ARSG2szBYD0ncWg], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T03:23:01,531][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]], allocation id [LZ5j5lUeQtOB03Wti27Bkw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T03:31:22,641][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [es-m02-rm] failed to execute on node [xmCfM_q6Saqf-xCyiao3Uw]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [es-d03-rm][192.168.0.153:9300][cluster:monitor/nodes/stats[n]] request_id [10048889] timed out after [15003ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:982) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
[2018-12-15T03:31:30,828][WARN ][o.e.t.TransportService ] [es-m02-rm] Received response for a request that has timed out, sent [23190ms] ago, timed out [8187ms] ago, action [cluster:monitor/nodes/stats[n]], node [{es-d03-rm}{xmCfM_q6Saqf-xCyiao3Uw}{5gHKeAW4SKehFhulHO_frA}{192.168.0.153}{192.168.0.153:9300}{faultDomain=0, updateDomain=2}], id [10048889]
[2018-12-15T03:39:34,485][INFO ][o.e.c.m.MetaDataMappingService] [es-m02-rm] [workitem_workitemcontract_1421_shared_c52fc8c9-8d1a-43f0-b2fe-325566cc3285/NvrqNUCfQZeOCiMUxJ7aIw] update_mapping [workItemContract]
[2018-12-15T03:56:36,170][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [es-m02-rm] failed to execute on node [05PeyBBySq-qL0NVHwdVmw]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [es-d01-rm][192.168.0.151:9300][cluster:monitor/nodes/stats[n]] request_id [10073816] timed out after [15007ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:982) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
[2018-12-15T03:56:59,040][WARN ][o.e.t.TransportService ] [es-m02-rm] Received response for a request that has timed out, sent [37877ms] ago, timed out [22870ms] ago, action [cluster:monitor/nodes/stats[n]], node [{es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0}], id [10073816]
[2018-12-15T03:58:07,652][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [es-m02-rm] failed to execute on node [p20g3NziScCwW9QaczvXZA]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [es-d02-rm][192.168.0.152:9300][cluster:monitor/nodes/stats[n]] request_id [10075330] timed out after [15016ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:982) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
[2018-12-15T03:58:20,725][WARN ][o.e.t.TransportService ] [es-m02-rm] Received response for a request that has timed out, sent [28089ms] ago, timed out [13073ms] ago, action [cluster:monitor/nodes/stats[n]], node [{es-d02-rm}{p20g3NziScCwW9QaczvXZA}{EHLNQB1ZSRSLQssKApJF8Q}{192.168.0.152}{192.168.0.152:9300}{faultDomain=1, updateDomain=1}], id [10075330]
[2018-12-15T04:23:02,636][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]], allocation id [tqoh_2k-TQKKBhHsB1SoXQ], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T05:23:03,755][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]], allocation id [vE8kWjY3Q-a99rsHkyvTLA], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T06:14:12,590][INFO ][o.e.c.s.ClusterSettings ] [es-m02-rm] updating [cluster.routing.allocation.cluster_concurrent_rebalance] from [2] to [10]
[2018-12-15T06:23:04,720][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]], allocation id [Xw_OY7HyQ5Kh8Ch1m6FE8w], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T07:23:05,737][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]], allocation id [YaIHeNnxTk2E6SuV-i5aag], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T08:23:06,798][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]], allocation id [iMa9lrasT-KamAHVslEO5A], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T09:23:07,914][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]], allocation id [BL0oHiDwR2OiTUCUEji1SA], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T09:47:44,673][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [codesearchshared_1_0][44] received shard failed for shard id [[codesearchshared_1_0][44]], allocation id [T-Cfsp_2SkeGgfjtZhfNKg], primary term [18], message [mark copy as stale]
[2018-12-15T10:02:13,011][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [codesearchshared_3_0][45] received shard failed for shard id [[codesearchshared_3_0][45]], allocation id [Mmfo3AJsT-y3WEVdRRddQA], primary term [16], message [mark copy as stale]
[2018-12-15T10:23:08,966][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]], allocation id [kkwLEf9wTRetY9AfZ2_yKA], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T11:23:10,013][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]], allocation id [kJZEUFjfTB6ulr4pDb61Jg], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T12:23:11,103][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]], allocation id [Mg-G2pFyQbSSCBH038aw8w], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T12:56:10,191][INFO ][o.e.c.m.MetaDataMappingService] [es-m02-rm] [workitemsearchshared_6_4/eqNH6WwjTlSXWtZBZG08Ng] update_mapping [workItemContract]
[2018-12-15T13:23:12,193][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]], allocation id [ugeU__1YQ4-eY1cjnsP_8Q], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T13:38:29,500][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [codesearchshared_0_0][11] received shard failed for shard id [[codesearchshared_0_0][11]], allocation id [JafPUU0nR0yLI6xHTLlhsw], primary term [14], message [mark copy as stale]
[2018-12-15T14:23:13,322][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]], allocation id [v1AYDXTQS3yvXLQqi0Dieg], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T14:49:27,068][INFO ][o.e.c.m.MetaDataMappingService] [es-m02-rm] [workitem_workitemcontract_1421_shared_20b77911-3fc0-4a96-a16f-5b3bbcd5ebf1/N3_qDWRxRQ2mbFp6gEKZbQ] update_mapping [workItemContract]
[2018-12-15T14:49:27,979][INFO ][o.e.c.m.MetaDataMappingService] [es-m02-rm] [workitem_workitemcontract_1421_shared_20b77911-3fc0-4a96-a16f-5b3bbcd5ebf1/N3_qDWRxRQ2mbFp6gEKZbQ] update_mapping [workItemContract]
[2018-12-15T14:49:28,070][INFO ][o.e.c.m.MetaDataMappingService] [es-m02-rm] [workitem_workitemcontract_1421_shared_20b77911-3fc0-4a96-a16f-5b3bbcd5ebf1/N3_qDWRxRQ2mbFp6gEKZbQ] update_mapping [workItemContract]
[2018-12-15T15:23:14,446][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]], allocation id [3lBopAgpRlW2bv-R-eyxlQ], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T15:44:31,471][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_88979bb0-32b6-4246-8039-e54f2d3c9113][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_88979bb0-32b6-4246-8039-e54f2d3c9113][1]], allocation id [83Lf8TasTJyFyD0DGgFJjQ], primary term [4], message [mark copy as stale]
[2018-12-15T15:51:26,880][INFO ][o.e.c.m.MetaDataMappingService] [es-m02-rm] [workitem_workitemcontract_1421_shared_20b77911-3fc0-4a96-a16f-5b3bbcd5ebf1/N3_qDWRxRQ2mbFp6gEKZbQ] update_mapping [workItemContract]
[2018-12-15T15:51:27,506][INFO ][o.e.c.m.MetaDataMappingService] [es-m02-rm] [workitem_workitemcontract_1421_shared_20b77911-3fc0-4a96-a16f-5b3bbcd5ebf1/N3_qDWRxRQ2mbFp6gEKZbQ] update_mapping [workItemContract]
[2018-12-15T15:57:12,270][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [workitem_workitemcontract_1421_shared_20b77911-3fc0-4a96-a16f-5b3bbcd5ebf1][4] received shard failed for shard id [[workitem_workitemcontract_1421_shared_20b77911-3fc0-4a96-a16f-5b3bbcd5ebf1][4]], allocation id [svLuVnlyS2qzFMscBB657Q], primary term [4], message [mark copy as stale]
[2018-12-15T16:23:15,537][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]], allocation id [emmL3NzlSsGp39cH44Ra6A], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T17:23:16,668][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]], allocation id [QCrxk2AqSuGc7RnuY1Z-3w], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T18:23:17,783][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]], allocation id [76u_CqZGQkyycYsLp6woZg], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T18:52:41,447][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_7f763e5c-d787-4a89-bef7-0bffc01c8d11][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_7f763e5c-d787-4a89-bef7-0bffc01c8d11][1]], allocation id [Y3xMBi4cT2aQnR72gthQRg], primary term [3], message [mark copy as stale]
[2018-12-15T19:23:18,750][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]], allocation id [UamoNIwjRwihg37-9FT2xQ], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T19:49:05,731][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [codesearchshared_6_0][94] received shard failed for shard id [[codesearchshared_6_0][94]], allocation id [OSCdRe_1Rpy_5J-r5UKScg], primary term [17], message [mark copy as stale]
[2018-12-15T20:23:19,914][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]], allocation id [GfOYI9ahT-iXhhO8Vz6CMw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T21:23:21,002][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]], allocation id [Y3krpsuoRBSvD1MvpR4fNQ], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d10-rm}{0pxsNqEkQ_Kel-CdOUrDqA}{zgSHBeeiTZOwA6PBrQG7zg}{192.168.0.160}{192.168.0.160:9300}{faultDomain=1, updateDomain=4} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T21:54:05,738][INFO ][o.e.c.m.MetaDataMappingService] [es-m02-rm] [workitemsearchshared_9_3/u9OKatYmSxG0MDmfociMGA] update_mapping [workItemContract]
[2018-12-15T22:05:20,728][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [codesearchshared_4_0][95] received shard failed for shard id [[codesearchshared_4_0][95]], allocation id [QQD7Zfv3QmWuAEIZOve40g], primary term [19], message [mark copy as stale]
[2018-12-15T22:05:20,728][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [codesearchshared_4_0][95] received shard failed for shard id [[codesearchshared_4_0][95]], allocation id [_DDUN8TmQES8BnS0LvxEiw], primary term [19], message [mark copy as stale]
[2018-12-15T22:23:22,123][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]], allocation id [wIr5zdi2QOmf70vOGU9VnA], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][1]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
[2018-12-15T23:23:23,085][WARN ][o.e.c.a.s.ShardStateAction] [es-m02-rm] [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0] received shard failed for shard id [[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]], allocation id [dD6_osylSF6dWCprWzZKsA], primary term [0], message [failed recovery], failure [RecoveryFailedException[[wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])]; nested: ElasticsearchTimeoutException[no activity after [30m]]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [wiki_wikicontract_1421_shared_e9529131-b634-4873-b279-c9dd8a0bd2e5][0]: Recovery failed from {es-d01-rm}{05PeyBBySq-qL0NVHwdVmw}{fGTsu8lHRNyuDlUglbjAlg}{192.168.0.151}{192.168.0.151:9300}{faultDomain=0, updateDomain=0} into {es-d08-rm}{EeSbzForSNeuewm02TeSlA}{7YAg4GuxRBORgeZ2ND0nDg}{192.168.0.158}{192.168.0.158:9300}{faultDomain=1, updateDomain=2} (no activity after [30m])
at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:286) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: no activity after [30m]
... 6 more
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment