Skip to content

Instantly share code, notes, and snippets.

@danielschonfeld
Created July 29, 2014 20:51
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save danielschonfeld/ff6c3744197f2c748632 to your computer and use it in GitHub Desktop.
Save danielschonfeld/ff6c3744197f2c748632 to your computer and use it in GitHub Desktop.
2014-07-29 20:50:10
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.55-b03 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00007f25c0007000 nid=0xf12 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"elasticsearch[betty][[socialrank-prod-v3][2]: Lucene Merge Thread #1026]" daemon prio=10 tid=0x00007f2580087000 nid=0xef0 waiting on condition [0x00007f247f835000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000058b727848> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at org.elasticsearch.search.SearchService$NormsWarmer$2.awaitTermination(SearchService.java:763)
at org.elasticsearch.indices.warmer.InternalIndicesWarmer.warmInternal(InternalIndicesWarmer.java:115)
at org.elasticsearch.indices.warmer.InternalIndicesWarmer.warm(InternalIndicesWarmer.java:71)
at org.elasticsearch.index.engine.internal.InternalEngine$2.warm(InternalEngine.java:1279)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4261)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3728)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:106)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
"Keep-Alive-Timer" daemon prio=10 tid=0x00007f25c4bdd800 nid=0xec8 waiting on condition [0x00007f244dea4000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at sun.net.www.http.KeepAliveCache.run(KeepAliveCache.java:172)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][generic][T#109]" daemon prio=10 tid=0x00007f25a41a2000 nid=0xeb5 waiting on condition [0x00007f247f433000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfac4d48> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"TimeLimitedCollector timer thread" daemon prio=10 tid=0x0000000001818000 nid=0xe4f waiting on condition [0x00007f2573a9d000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.lucene.search.TimeLimitingCollector$TimerThread.run(TimeLimitingCollector.java:267)
"elasticsearch[betty][refresh][T#9]" daemon prio=10 tid=0x00007f25a419e000 nid=0xdb0 waiting on condition [0x00007f247f31b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004f60ad9c0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at org.apache.lucene.index.DocumentsWriterFlushControl.markForFullFlush(DocumentsWriterFlushControl.java:481)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:617)
- locked <0x000000050043cb58> (a org.apache.lucene.index.DocumentsWriter)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:373)
- locked <0x00000004dae60328> (a java.lang.Object)
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:287)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:252)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:171)
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:118)
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
at org.elasticsearch.index.engine.internal.InternalEngine.refresh(InternalEngine.java:715)
- locked <0x00000004f566b938> (a java.lang.Object)
at org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:479)
at org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:928)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][flush][T#5]" daemon prio=10 tid=0x00007f25a419e800 nid=0xd47 in Object.wait() [0x00007f2573895000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000004ea7b69f0> (a org.apache.lucene.index.DocumentsWriterFlushControl)
at java.lang.Object.wait(Object.java:503)
at org.apache.lucene.index.DocumentsWriterFlushControl.waitForFlush(DocumentsWriterFlushControl.java:251)
- locked <0x00000004ea7b69f0> (a org.apache.lucene.index.DocumentsWriterFlushControl)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:631)
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2937)
- locked <0x00000004dae6b330> (a java.lang.Object)
- locked <0x00000004dae6b318> (a java.lang.Object)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3096)
- locked <0x00000004dae6b318> (a java.lang.Object)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3063)
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:797)
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:604)
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:202)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][merge][T#9]" daemon prio=10 tid=0x00007f25a4226000 nid=0xd43 waiting on condition [0x00007f247f93b000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfca2e00> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:731)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1145)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][warmer][T#8]" daemon prio=10 tid=0x00007f25cc00a000 nid=0xd42 runnable [0x00007f25738d5000]
java.lang.Thread.State: RUNNABLE
at org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.scanToTermLeaf(BlockTreeTermsReader.java:2834)
at org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.scanToTerm(BlockTreeTermsReader.java:2724)
at org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1736)
at org.apache.lucene.queries.TermsFilter.getDocIdSet(TermsFilter.java:196)
at org.elasticsearch.index.cache.filter.support.CacheKeyFilter$Wrapper.getDocIdSet(CacheKeyFilter.java:91)
at org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:178)
at org.elasticsearch.common.lucene.search.XBooleanFilter.getDocIdSet(XBooleanFilter.java:59)
at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45)
at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:157)
at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
at org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
at org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149)
at org.elasticsearch.search.SearchService$SearchWarmer$1.run(SearchService.java:899)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][merge][T#8]" daemon prio=10 tid=0x00007f25a4001800 nid=0xd39 waiting on condition [0x00007f2573854000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfca2e00> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:731)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1145)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][warmer][T#7]" daemon prio=10 tid=0x00007f257c027000 nid=0xd37 runnable [0x00007f247f8fa000]
java.lang.Thread.State: RUNNABLE
at org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1722)
at org.apache.lucene.queries.TermsFilter.getDocIdSet(TermsFilter.java:196)
at org.elasticsearch.index.cache.filter.support.CacheKeyFilter$Wrapper.getDocIdSet(CacheKeyFilter.java:91)
at org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:178)
at org.elasticsearch.common.lucene.search.XBooleanFilter.getDocIdSet(XBooleanFilter.java:59)
at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45)
at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:157)
at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
at org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
at org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:175)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:149)
at org.elasticsearch.search.SearchService$SearchWarmer$1.run(SearchService.java:899)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][flush][T#4]" daemon prio=10 tid=0x00007f25a419d000 nid=0xcda waiting for monitor entry [0x00007f23c1189000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.elasticsearch.index.engine.internal.InternalEngine.refresh(InternalEngine.java:710)
- waiting to lock <0x00000004f566b938> (a java.lang.Object)
at org.elasticsearch.index.engine.internal.InternalEngine.refreshVersioningTable(InternalEngine.java:870)
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:798)
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:604)
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:202)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][refresh][T#4]" daemon prio=10 tid=0x00007f25a41a0000 nid=0xb17 waiting on condition [0x00007f247f701000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000052d4505c0> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at org.elasticsearch.search.SearchService$SearchWarmer$2.awaitTermination(SearchService.java:923)
at org.elasticsearch.indices.warmer.InternalIndicesWarmer.warmInternal(InternalIndicesWarmer.java:115)
at org.elasticsearch.indices.warmer.InternalIndicesWarmer.warm(InternalIndicesWarmer.java:71)
at org.elasticsearch.index.engine.internal.InternalEngine$SearchFactory.newSearcher(InternalEngine.java:1494)
at org.apache.lucene.search.SearcherManager.getSearcher(SearcherManager.java:160)
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:122)
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
at org.elasticsearch.index.engine.internal.InternalEngine.refresh(InternalEngine.java:715)
- locked <0x00000004f5641848> (a java.lang.Object)
at org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:479)
at org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:928)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][index][T#4]" daemon prio=10 tid=0x00007f2594086800 nid=0xb6 runnable [0x00007f245c936000]
java.lang.Thread.State: RUNNABLE
at org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser.parseMediumName(UTF8StreamJsonParser.java:1575)
at org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1511)
at org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:700)
at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:50)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:475)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:538)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:615)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:482)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:515)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:462)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:394)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:194)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:534)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.doStart(TransportShardReplicationOperationAction.java:440)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.start(TransportShardReplicationOperationAction.java:338)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction.doExecute(TransportShardReplicationOperationAction.java:104)
at org.elasticsearch.action.index.TransportIndexAction.innerExecute(TransportIndexAction.java:125)
at org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:107)
at org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:60)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:230)
at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:185)
at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:64)
at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.run(TransportInstanceSingleOperationAction.java:193)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][index][T#3]" daemon prio=10 tid=0x00007f258c225000 nid=0xb4 runnable [0x00007f247f8b8000]
java.lang.Thread.State: RUNNABLE
at org.apache.lucene.index.TermVectorsConsumerPerField.start(TermVectorsConsumerPerField.java:70)
at org.apache.lucene.index.TermsHashPerField.start(TermsHashPerField.java:129)
at org.apache.lucene.index.TermsHashPerField.start(TermsHashPerField.java:132)
at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:68)
at org.apache.lucene.index.DocFieldProcessor.processDocument(DocFieldProcessor.java:248)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:298)
at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:427)
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1306)
at org.elasticsearch.index.engine.internal.InternalEngine.innerIndex(InternalEngine.java:529)
- locked <0x00000004f566a258> (a java.lang.Object)
at org.elasticsearch.index.engine.internal.InternalEngine.index(InternalEngine.java:477)
at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:406)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnReplica(TransportIndexAction.java:239)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:249)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:228)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][index][T#2]" daemon prio=10 tid=0x00007f258410b000 nid=0xb3 runnable [0x00007f244edca000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.hash(HashMap.java:366)
at java.util.HashMap.put(HashMap.java:496)
at org.elasticsearch.index.mapper.ParseContext.ignoredValue(ParseContext.java:326)
at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:298)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:407)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:637)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:490)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:538)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:615)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:482)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:515)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:462)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:394)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:194)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:534)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.doStart(TransportShardReplicationOperationAction.java:440)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.start(TransportShardReplicationOperationAction.java:338)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction.doExecute(TransportShardReplicationOperationAction.java:104)
at org.elasticsearch.action.index.TransportIndexAction.innerExecute(TransportIndexAction.java:125)
at org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:107)
at org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:60)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:230)
at org.elasticsearch.action.update.TransportUpdateAction$3$1.run(TransportUpdateAction.java:246)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][index][T#1]" daemon prio=10 tid=0x00007f259c112000 nid=0xaf runnable [0x00007f25705ed000]
java.lang.Thread.State: RUNNABLE
at org.apache.lucene.index.StoredFieldsProcessor.fill(StoredFieldsProcessor.java:110)
at org.apache.lucene.index.StoredFieldsProcessor.finishDocument(StoredFieldsProcessor.java:121)
at org.apache.lucene.index.TwoStoredFieldsConsumers.finishDocument(TwoStoredFieldsConsumers.java:65)
at org.apache.lucene.index.DocFieldProcessor.finishDocument(DocFieldProcessor.java:262)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:314)
at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:427)
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1306)
at org.elasticsearch.index.engine.internal.InternalEngine.innerIndex(InternalEngine.java:529)
- locked <0x00000004f5501d08> (a java.lang.Object)
at org.elasticsearch.index.engine.internal.InternalEngine.index(InternalEngine.java:477)
at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:406)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnReplica(TransportIndexAction.java:239)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:249)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:228)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][get][T#4]" daemon prio=10 tid=0x00007f258c107000 nid=0x88 waiting on condition [0x00007f247f9bd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfca01a0> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][get][T#3]" daemon prio=10 tid=0x00007f2594090000 nid=0x87 waiting on condition [0x00007f2573f3b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfca01a0> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][get][T#2]" daemon prio=10 tid=0x00007f2590113000 nid=0x86 waiting on condition [0x00007f247f877000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfca01a0> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][get][T#1]" daemon prio=10 tid=0x00007f2584108000 nid=0x85 runnable [0x00007f2573812000]
java.lang.Thread.State: RUNNABLE
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.intValue(AbstractXContentParser.java:117)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:297)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:275)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:308)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readList(AbstractXContentParser.java:284)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:310)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:275)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readOrderedMap(AbstractXContentParser.java:258)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.mapOrdered(AbstractXContentParser.java:213)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.mapOrderedAndClose(AbstractXContentParser.java:228)
at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:119)
at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:75)
at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:297)
at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:106)
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:109)
at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:43)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:263)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$ShardTransportHandler.messageReceived(TransportShardSingleOperationAction.java:249)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][bulk][T#4]" daemon prio=10 tid=0x00007f2584106800 nid=0x84 waiting for monitor entry [0x00007f24664dd000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.elasticsearch.index.engine.internal.InternalEngine.innerIndex(InternalEngine.java:489)
- waiting to lock <0x00000004f5663dc0> (a java.lang.Object)
at org.elasticsearch.index.engine.internal.InternalEngine.index(InternalEngine.java:477)
at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:406)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:420)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardUpdateOperation(TransportShardBulkAction.java:522)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:236)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:534)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:433)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][bulk][T#3]" daemon prio=10 tid=0x00007f2588104000 nid=0x83 waiting for monitor entry [0x00007f246651e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.elasticsearch.index.engine.internal.InternalEngine.innerCreate(InternalEngine.java:404)
- waiting to lock <0x00000004f5501d08> (a java.lang.Object)
at org.elasticsearch.index.engine.internal.InternalEngine.create(InternalEngine.java:386)
at org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:384)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:575)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:249)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:228)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][bulk][T#2]" daemon prio=10 tid=0x00007f2590174800 nid=0x82 runnable [0x00007f245a348000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.hash(HashMap.java:366)
at java.util.HashMap.put(HashMap.java:496)
at java.util.HashSet.add(HashSet.java:217)
at org.apache.lucene.codecs.lucene42.Lucene42NormsConsumer.addNumericField(Lucene42NormsConsumer.java:108)
at org.apache.lucene.index.NumericDocValuesWriter.flush(NumericDocValuesWriter.java:91)
at org.apache.lucene.index.NormsConsumerPerField.flush(NormsConsumerPerField.java:61)
at org.apache.lucene.index.NormsConsumer.flush(NormsConsumer.java:56)
at org.apache.lucene.index.DocInverter.flush(DocInverter.java:54)
at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:81)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:465)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:518)
at org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:368)
at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:412)
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1306)
at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1267)
at org.elasticsearch.index.engine.internal.InternalEngine.innerCreate(InternalEngine.java:457)
- locked <0x00000004f5663dc0> (a java.lang.Object)
at org.elasticsearch.index.engine.internal.InternalEngine.create(InternalEngine.java:386)
at org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:384)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:430)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardUpdateOperation(TransportShardBulkAction.java:522)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:236)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:534)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:433)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][bulk][T#1]" daemon prio=10 tid=0x00007f2584106000 nid=0x81 waiting for monitor entry [0x00007f245a38a000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.elasticsearch.index.engine.internal.InternalEngine.innerCreate(InternalEngine.java:404)
- waiting to lock <0x00000004f566a258> (a java.lang.Object)
at org.elasticsearch.index.engine.internal.InternalEngine.create(InternalEngine.java:386)
at org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:384)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:575)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:249)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:228)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][management][T#5]" daemon prio=10 tid=0x00007f259408e000 nid=0x6f waiting on condition [0x00007f247f742000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfac3f48> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:731)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1145)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][search][T#12]" daemon prio=10 tid=0x00007f2590110000 nid=0x6e waiting on condition [0x00007f247ad7b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfc9d678> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][search][T#11]" daemon prio=10 tid=0x00007f259c116000 nid=0x6d waiting on condition [0x00007f247ad38000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfc9d678> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][search][T#10]" daemon prio=10 tid=0x00007f259010e800 nid=0x6c waiting on condition [0x00007f247a4ed000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfc9d678> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][search][T#9]" daemon prio=10 tid=0x00007f259010d000 nid=0x6b waiting on condition [0x00007f247a52e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfc9d678> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][search][T#8]" daemon prio=10 tid=0x00007f25a0083800 nid=0x6a waiting on condition [0x00007f247a56f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfc9d678> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][search][T#7]" daemon prio=10 tid=0x00007f259010b800 nid=0x69 waiting on condition [0x00007f247a752000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfc9d678> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][search][T#6]" daemon prio=10 tid=0x00007f2590109800 nid=0x68 waiting on condition [0x00007f247a793000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfc9d678> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][search][T#5]" daemon prio=10 tid=0x00007f2590107800 nid=0x67 waiting on condition [0x00007f247a7d4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfc9d678> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][search][T#4]" daemon prio=10 tid=0x00007f258c083800 nid=0x66 waiting on condition [0x00007f247a83a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfc9d678> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][search][T#3]" daemon prio=10 tid=0x00007f259c114000 nid=0x65 waiting on condition [0x00007f247a87b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfc9d678> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][search][T#2]" daemon prio=10 tid=0x00007f2590104000 nid=0x5f waiting on condition [0x00007f247f97c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfc9d678> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][management][T#4]" daemon prio=10 tid=0x00007f259408d000 nid=0x5d waiting on condition [0x00007f2573917000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfac3f48> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:731)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1145)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][management][T#3]" daemon prio=10 tid=0x00007f2598102800 nid=0x57 waiting on condition [0x00007f251af1b000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfac3f48> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:731)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1145)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][search][T#1]" daemon prio=10 tid=0x00007f25a0088000 nid=0x56 waiting on condition [0x00007f251af5c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfc9d678> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][management][T#2]" daemon prio=10 tid=0x00007f2598101000 nid=0x55 waiting on condition [0x00007f25704e9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfac3f48> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:731)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1145)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][management][T#1]" daemon prio=10 tid=0x00007f2590081000 nid=0x4f waiting on condition [0x00007f2573999000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfac3f48> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:731)
at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1145)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][keep_alive]" daemon prio=10 tid=0x00007f25a8002800 nid=0x4e waiting on condition [0x00007f2573a1b000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.elasticsearch.marvel.agent.exporter.ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:705)
at java.lang.Thread.run(Thread.java:745)
"DestroyJavaVM" prio=10 tid=0x00007f260400a800 nid=0x11 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"elasticsearch[keepAlive/1.2.2]" prio=10 tid=0x00007f2605028000 nid=0x4d waiting on condition [0x00007f2573a5c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004d0303cb8> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at org.elasticsearch.bootstrap.Bootstrap$3.run(Bootstrap.java:225)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][clusterService#updateTask][T#1]" daemon prio=10 tid=0x00007f25a008c800 nid=0x4c waiting on condition [0x00007f25739da000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfac0e70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:539)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_client_timer][T#1]{Hashed wheel timer #1}" daemon prio=10 tid=0x00007f257c00a000 nid=0x46 waiting on condition [0x00007f2573958000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:483)
at org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:392)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_server_boss][T#1]{New I/O server boss #27}" daemon prio=10 tid=0x00007f2605024000 nid=0x3e runnable [0x00007f2573f7c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004d0068358> (a sun.nio.ch.Util$2)
- locked <0x00000004d0068370> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfd4aa98> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_server_worker][T#8]{New I/O worker #26}" daemon prio=10 tid=0x00007f2604ff9000 nid=0x3d runnable [0x00007f2573fbd000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004d0335f00> (a sun.nio.ch.Util$2)
- locked <0x00000004d0841f88> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfc972a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_server_worker][T#7]{New I/O worker #25}" daemon prio=10 tid=0x00007f2604fcd800 nid=0x3c runnable [0x00007f2573ffe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004d02f2778> (a sun.nio.ch.Util$2)
- locked <0x00000004d02f2790> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfd82ca8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_server_worker][T#6]{New I/O worker #24}" daemon prio=10 tid=0x00007f2604fa2800 nid=0x3b runnable [0x00007f25b4079000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfac4d78> (a sun.nio.ch.Util$2)
- locked <0x00000004cfac4d90> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfac9160> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_server_worker][T#5]{New I/O worker #23}" daemon prio=10 tid=0x00007f2604ea1000 nid=0x3a runnable [0x00007f25b40ba000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfac4e98> (a sun.nio.ch.Util$2)
- locked <0x00000004cfac4e80> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfac91f0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_server_worker][T#4]{New I/O worker #22}" daemon prio=10 tid=0x00007f2604e9f000 nid=0x39 runnable [0x00007f25b40fb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfae0cc8> (a sun.nio.ch.Util$2)
- locked <0x00000004cfae0ce0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfadfc78> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_server_worker][T#3]{New I/O worker #21}" daemon prio=10 tid=0x00007f2604e99000 nid=0x38 runnable [0x00007f25b413c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfae0d58> (a sun.nio.ch.Util$2)
- locked <0x00000004cfae0d70> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfadfd08> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_server_worker][T#2]{New I/O worker #20}" daemon prio=10 tid=0x00007f2604e97800 nid=0x37 runnable [0x00007f25b417d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfae4fd0> (a sun.nio.ch.Util$2)
- locked <0x00000004cfae4fb8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfadfd50> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_server_worker][T#1]{New I/O worker #19}" daemon prio=10 tid=0x00007f2604e9a800 nid=0x36 runnable [0x00007f25b41be000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfb08580> (a sun.nio.ch.Util$2)
- locked <0x00000004cfac96b8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfac8918> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_client_boss][T#1]{New I/O boss #18}" daemon prio=10 tid=0x00007f2604e9d000 nid=0x35 runnable [0x00007f25b41ff000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfb21dd0> (a sun.nio.ch.Util$2)
- locked <0x00000004cfb21db8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfac89a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_client_worker][T#8]{New I/O worker #17}" daemon prio=10 tid=0x00007f2604e80000 nid=0x34 runnable [0x00007f25b4240000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfb37008> (a sun.nio.ch.Util$2)
- locked <0x00000004cfb36ff0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfac8a80> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_client_worker][T#7]{New I/O worker #16}" daemon prio=10 tid=0x00007f2604e55000 nid=0x33 runnable [0x00007f25b4281000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfb8bee8> (a sun.nio.ch.Util$2)
- locked <0x00000004cfb8bed0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfac8d08> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_client_worker][T#6]{New I/O worker #15}" daemon prio=10 tid=0x00007f2604e2a000 nid=0x32 runnable [0x00007f25b42c2000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfb61818> (a sun.nio.ch.Util$2)
- locked <0x00000004cfb61800> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfac8be8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_client_worker][T#5]{New I/O worker #14}" daemon prio=10 tid=0x00007f2604dff000 nid=0x31 runnable [0x00007f25b4303000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfb75fd0> (a sun.nio.ch.Util$2)
- locked <0x00000004cfb75fb8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfac8c78> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_client_worker][T#4]{New I/O worker #13}" daemon prio=10 tid=0x00007f2604dd4000 nid=0x30 runnable [0x00007f25b4344000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfbb74d8> (a sun.nio.ch.Util$2)
- locked <0x00000004cfbb74c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfb8cc30> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_client_worker][T#3]{New I/O worker #12}" daemon prio=10 tid=0x00007f2604d3d800 nid=0x2f runnable [0x00007f25b4385000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfba0898> (a sun.nio.ch.Util$2)
- locked <0x00000004cfba0880> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfb8cba0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_client_worker][T#2]{New I/O worker #11}" daemon prio=10 tid=0x00007f2604d3c000 nid=0x2e runnable [0x00007f25b43c6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfb4c0d8> (a sun.nio.ch.Util$2)
- locked <0x00000004cfb4c0c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfac8b58> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][transport_client_worker][T#1]{New I/O worker #10}" daemon prio=10 tid=0x00007f2604d3b000 nid=0x2d runnable [0x00007f25b4407000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfbcb968> (a sun.nio.ch.Util$2)
- locked <0x00000004cfbcb950> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfb8ccc0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][[ttl_expire]]" daemon prio=10 tid=0x00007f2604d39800 nid=0x2c waiting on condition [0x00007f25b4448000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfbe16d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2176)
at org.elasticsearch.indices.ttl.IndicesTTLService$Notifier.await(IndicesTTLService.java:325)
at org.elasticsearch.indices.ttl.IndicesTTLService$PurgerThread.run(IndicesTTLService.java:147)
"elasticsearch[betty][http_server_boss][T#1]{New I/O server boss #9}" daemon prio=10 tid=0x00007f2604b4a000 nid=0x2b runnable [0x00007f25b48af000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfbede10> (a sun.nio.ch.Util$2)
- locked <0x00000004cfbeddf8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfb4f228> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][http_server_worker][T#8]{New I/O worker #8}" daemon prio=10 tid=0x00007f2604b45000 nid=0x2a runnable [0x00007f25b48f0000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfcb57c8> (a sun.nio.ch.Util$2)
- locked <0x00000004cfcb57b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfae0020> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][http_server_worker][T#7]{New I/O worker #7}" daemon prio=10 tid=0x00007f2604b43000 nid=0x29 runnable [0x00007f25b4931000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfd1fe98> (a sun.nio.ch.Util$2)
- locked <0x00000004cfd1fe80> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfcc6c40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][http_server_worker][T#6]{New I/O worker #6}" daemon prio=10 tid=0x00007f2604b41000 nid=0x28 runnable [0x00007f25b4972000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfcfda28> (a sun.nio.ch.Util$2)
- locked <0x00000004cfcfda10> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfcc6b20> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][http_server_worker][T#5]{New I/O worker #5}" daemon prio=10 tid=0x00007f2604b3f800 nid=0x27 runnable [0x00007f25b49b3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfd30d60> (a sun.nio.ch.Util$2)
- locked <0x00000004cfd30d48> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfcc6cd0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][http_server_worker][T#4]{New I/O worker #4}" daemon prio=10 tid=0x00007f2604b3e000 nid=0x26 runnable [0x00007f25b49f4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfcc7018> (a sun.nio.ch.Util$2)
- locked <0x00000004cfcc7000> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfcc6970> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][http_server_worker][T#3]{New I/O worker #3}" daemon prio=10 tid=0x00007f2604aca000 nid=0x25 runnable [0x00007f25b4a35000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfcd83b8> (a sun.nio.ch.Util$2)
- locked <0x00000004cfcd83a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfcc6a90> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][http_server_worker][T#2]{New I/O worker #2}" daemon prio=10 tid=0x00007f2604ac9800 nid=0x24 runnable [0x00007f25b4a76000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfcfdb30> (a sun.nio.ch.Util$2)
- locked <0x00000004cfcfdb18> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfcc6bb0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][http_server_worker][T#1]{New I/O worker #1}" daemon prio=10 tid=0x00007f2604ac8800 nid=0x23 runnable [0x00007f25b4ab7000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000004cfcd82c8> (a sun.nio.ch.Util$2)
- locked <0x00000004cfcd82b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000004cfcc6a00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][marvel.exporters]" daemon prio=10 tid=0x00007f2604abd000 nid=0x22 waiting on condition [0x00007f25b4af8000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:230)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][scheduler][T#1]" daemon prio=10 tid=0x00007f2604a81800 nid=0x21 waiting on condition [0x00007f25b4b39000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004cfc6cd60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"elasticsearch[betty][[timer]]" daemon prio=10 tid=0x00007f26049bf000 nid=0x20 waiting on condition [0x00007f25b4b7a000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.elasticsearch.threadpool.ThreadPool$EstimatedTimeThread.run(ThreadPool.java:518)
"Service Thread" daemon prio=10 tid=0x00007f2604112800 nid=0x1e runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007f260410f800 nid=0x1d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007f260410d800 nid=0x1c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f260410b000 nid=0x1b runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007f2604101000 nid=0x1a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007f26040eb800 nid=0x19 in Object.wait() [0x00007f25dcaa6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000004cfd4e508> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000004cfd4e508> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
"Reference Handler" daemon prio=10 tid=0x00007f26040e7800 nid=0x18 in Object.wait() [0x00007f25dcae7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000004cfd52cc8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000004cfd52cc8> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x00007f26040e5000 nid=0x17 runnable
"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f260401c000 nid=0x12 runnable
"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f260401d800 nid=0x13 runnable
"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f260401f800 nid=0x14 runnable
"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f2604021800 nid=0x15 runnable
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f26040a1800 nid=0x16 runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f260411d800 nid=0x1f waiting on condition
JNI global references: 335
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment