Skip to content

Instantly share code, notes, and snippets.

@bryanhelmig

bryanhelmig/gist:3c17edfe5c4e9065e5a3 Secret

Created December 17, 2013 03:56
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bryanhelmig/3c17edfe5c4e9065e5a3 to your computer and use it in GitHub Desktop.
Save bryanhelmig/3c17edfe5c4e9065e5a3 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
ubuntu@domU-12-31-39-0A-04-E1:/tmp$ cat /var/log/elasticsearch/Rage\ Against\ the\ Machine.log | head -n 100
[2013-12-17 02:08:04,236][WARN ][cluster.action.shard ] [Ardroman] [zapier_legacy][0] received shard failed for [zapier_legacy][0], node[vJhywv0iSEmn057Ccs-fSQ], [P], s[STARTED], indexUUID [pzWL-WO_SsaGbuWfn2IQaw], reason [engine failure, message [MergeException[java.io.EOFException: read past EOF: NIOFSIndexInput(path="/var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/_2jf6.fdt")]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/_2jf6.fdt")]; ]]
[2013-12-17 02:10:45,311][WARN ][cluster.action.shard ] [Ardroman] [zapier_legacy][0] received shard failed for [zapier_legacy][0], node[vJhywv0iSEmn057Ccs-fSQ], relocating [GwtG-bJnRguXVoV5CmvgMA], [R], s[INITIALIZING], indexUUID [pzWL-WO_SsaGbuWfn2IQaw], reason [Failed to start shard, message [RecoveryFailedException[[zapier_legacy][0]: Recovery failed from [Calm Mantis][wdsXL2wpSruxqyVJNajGaQ][inet[/10.147.132.135:9300]]{aws_availability_zone=us-east-1d} into [Carnivore][vJhywv0iSEmn057Ccs-fSQ][inet[/10.45.147.163:9300]]{aws_availability_zone=us-east-1b}]; nested: RemoteTransportException[[Calm Mantis][inet[/10.147.132.135:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[zapier_legacy][0] Phase[2] Execution failed]; nested: RemoteTransportException[[Carnivore][inet[/10.45.147.163:9300]][index/shard/recovery/prepareTranslog]]; nested: EngineCreationFailureException[[zapier_legacy][0] failed to create engine]; nested: LockObtainFailedException[Lock obtain timed out: NativeFSLock@/var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/write.lock]; ]]
[2013-12-17 02:55:56,585][WARN ][index.merge.scheduler ] [Ardroman] [zapier_legacy][0] failed to merge
java.io.EOFException: read past EOF: NIOFSIndexInput(path="/var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/_2jf6.fdt")
at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:156)
at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:113)
at org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:254)
at org.elasticsearch.index.store.Store$StoreIndexOutput.copyBytes(Store.java:604)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$ChunkIterator.copyCompressedData(CompressingStoredFieldsReader.java:495)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:375)
at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:316)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:94)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3855)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3458)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
[2013-12-17 02:55:56,588][WARN ][index.engine.robin ] [Ardroman] [zapier_legacy][0] failed engine
org.apache.lucene.index.MergePolicy$MergeException: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/_2jf6.fdt")
at org.elasticsearch.index.merge.scheduler.ConcurrentMergeSchedulerProvider$CustomConcurrentMergeScheduler.handleMergeException(ConcurrentMergeSchedulerProvider.java:109)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/_2jf6.fdt")
at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:156)
at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:113)
at org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:254)
at org.elasticsearch.index.store.Store$StoreIndexOutput.copyBytes(Store.java:604)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$ChunkIterator.copyCompressedData(CompressingStoredFieldsReader.java:495)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:375)
at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:316)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:94)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3855)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3458)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
[2013-12-17 02:55:56,591][WARN ][cluster.action.shard ] [Ardroman] [zapier_legacy][0] sending failed shard for [zapier_legacy][0], node[GwtG-bJnRguXVoV5CmvgMA], [R], s[STARTED], indexUUID [pzWL-WO_SsaGbuWfn2IQaw], reason [engine failure, message [MergeException[java.io.EOFException: read past EOF: NIOFSIndexInput(path="/var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/_2jf6.fdt")]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/_2jf6.fdt")]; ]]
[2013-12-17 02:55:56,591][WARN ][cluster.action.shard ] [Ardroman] [zapier_legacy][0] received shard failed for [zapier_legacy][0], node[GwtG-bJnRguXVoV5CmvgMA], [R], s[STARTED], indexUUID [pzWL-WO_SsaGbuWfn2IQaw], reason [engine failure, message [MergeException[java.io.EOFException: read past EOF: NIOFSIndexInput(path="/var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/_2jf6.fdt")]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/_2jf6.fdt")]; ]]
[2013-12-17 02:55:59,008][WARN ][cluster.action.shard ] [Ardroman] [zapier_legacy][0] received shard failed for [zapier_legacy][0], node[wdsXL2wpSruxqyVJNajGaQ], [P], s[STARTED], indexUUID [pzWL-WO_SsaGbuWfn2IQaw], reason [engine failure, message [MergeException[java.io.EOFException: read past EOF: NIOFSIndexInput(path="/var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/_2jf6.fdt")]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/_2jf6.fdt")]; ]]
[2013-12-17 02:55:59,044][WARN ][index.engine.robin ] [Ardroman] [zapier_legacy][0] shard is locked, releasing lock
[2013-12-17 02:55:59,045][WARN ][indices.cluster ] [Ardroman] [zapier_legacy][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [zapier_legacy][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:232)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [zapier_legacy][0] failed to create engine
at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:259)
at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:665)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:174)
... 3 more
Caused by: org.apache.lucene.store.LockReleaseFailedException: Cannot forcefully unlock a NativeFSLock which is held by another indexer component: /var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/write.lock
at org.apache.lucene.store.NativeFSLock.release(NativeFSLockFactory.java:295)
at org.apache.lucene.index.IndexWriter.unlock(IndexWriter.java:4256)
at org.elasticsearch.index.engine.robin.RobinEngine.createWriter(RobinEngine.java:1400)
at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:257)
... 6 more
[2013-12-17 02:55:59,046][WARN ][cluster.action.shard ] [Ardroman] [zapier_legacy][0] sending failed shard for [zapier_legacy][0], node[GwtG-bJnRguXVoV5CmvgMA], [P], s[INITIALIZING], indexUUID [pzWL-WO_SsaGbuWfn2IQaw], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[zapier_legacy][0] failed recovery]; nested: EngineCreationFailureException[[zapier_legacy][0] failed to create engine]; nested: LockReleaseFailedException[Cannot forcefully unlock a NativeFSLock which is held by another indexer component: /var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/write.lock]; ]]
[2013-12-17 02:55:59,046][WARN ][cluster.action.shard ] [Ardroman] [zapier_legacy][0] received shard failed for [zapier_legacy][0], node[GwtG-bJnRguXVoV5CmvgMA], [P], s[INITIALIZING], indexUUID [pzWL-WO_SsaGbuWfn2IQaw], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[zapier_legacy][0] failed recovery]; nested: EngineCreationFailureException[[zapier_legacy][0] failed to create engine]; nested: LockReleaseFailedException[Cannot forcefully unlock a NativeFSLock which is held by another indexer component: /var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/write.lock]; ]]
[2013-12-17 02:56:05,703][WARN ][index.engine.robin ] [Ardroman] [zapier_legacy][0] Could not lock IndexWriter isLocked [false]
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/data/elasticsearch/Rage Against the Machine/nodes/0/indices/zapier_legacy/0/index/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:695)
at org.apache.lucene.index.XIndexWriter.<init>(XIndexWriter.java:45)
at org.elasticsearch.index.engine.robin.RobinEngine.createWriter(RobinEngine.java:1443)
at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:257)
at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:665)
at org.elasticsearch.indices.recovery.RecoveryTarget$PrepareForTranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:389)
at org.elasticsearch.indices.recovery.RecoveryTarget$PrepareForTranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:363)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
[2013-12-17 02:56:05,715][WARN ][indices.cluster ] [Ardroman] [zapier_legacy][0] failed to start shard
org.elasticsearch.indices.recovery.RecoveryFailedException: [zapier_legacy][0]: Recovery failed from [Calm Mantis][wdsXL2wpSruxqyVJNajGaQ][inet[/10.147.132.135:9300]]{aws_availability_zone=us-east-1d} into [Ardroman][GwtG-bJnRguXVoV5CmvgMA][inet[/10.211.7.15:9300]]{aws_availability_zone=us-east-1c}
at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:303)
at org.elasticsearch.indices.recovery.RecoveryTarget.access$300(RecoveryTarget.java:65)
at org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:171)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.elasticsearch.transport.RemoteTransportException: [Calm Mantis][inet[/10.147.132.135:9300]][index/shard/recovery/startRecovery]
Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [zapier_legacy][0] Phase[2] Execution failed
at org.elasticsearch.index.engine.robin.RobinEngine.recover(RobinEngine.java:1156)
at org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:590)
at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:117)
at org.elasticsearch.indices.recovery.RecoverySource.access$1600(RecoverySource.java:61)
at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:333)
at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:319)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.elasticsearch.transport.RemoteTransportException: [Ardroman][inet[/10.211.7.15:9300]][index/shard/recovery/prepareTranslog]
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [zapier_legacy][0] failed to create engine
at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:259)
at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:665)
at org.elasticsearch.indices.recovery.RecoveryTarget$PrepareForTranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:389)
at org.elasticsearch.indices.recovery.RecoveryTarget$PrepareForTranslogOperationsRequestHandler.messageReceived(RecoveryTarget.java:363)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment