Skip to content

Instantly share code, notes, and snippets.

@travisbell

travisbell/jan-12-1.txt

Created January 13, 2016 15:29
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save travisbell/c08f10b928db9c538430 to your computer and use it in GitHub Desktop.
Save travisbell/c08f10b928db9c538430 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
jan-12-1.txt
::: [xx-host-1400][Y4qXPFeeRK2uKbggdiReig][xx-host-1400.domain.com][inet[/1400_ip:9300]]{aws_availability_zone=us-east-1d, master=true}
Hot threads at 2016-01-12T20:51:55.974Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
40.1% (200.4ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#6]'
7/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
38.7% (193.5ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#4]'
2/10 snapshots sharing following 33 elements
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:726)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
org.apache.lucene.store.DataInput.readVInt(DataInput.java:122)
org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:221)
org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:152)
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:506)
org.apache.lucene.index.TermContext.build(TermContext.java:95)
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:165)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.<init>(DisjunctionMaxQuery.java:123)
org.apache.lucene.search.DisjunctionMaxQuery.createWeight(DisjunctionMaxQuery.java:201)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery.createWeight(FunctionScoreQuery.java:95)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:684)
org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:133)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
5/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
::: [xx-host-1200][i0oiyUh4Tb-KtDUkOFzzXg][xx-host-1200.domain.com][inet[/1200_ip:9300]]{aws_availability_zone=us-east-1b, master=true}
Hot threads at 2016-01-12T20:51:55.953Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
25.1% (125.4ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#13]'
2/10 snapshots sharing following 26 elements
org.elasticsearch.index.search.MultiMatchQuery.access$000(MultiMatchQuery.java:44)
org.elasticsearch.index.search.MultiMatchQuery$QueryBuilder.parseGroup(MultiMatchQuery.java:120)
org.elasticsearch.index.search.MultiMatchQuery$QueryBuilder.buildGroupedQueries(MultiMatchQuery.java:111)
org.elasticsearch.index.search.MultiMatchQuery.parse(MultiMatchQuery.java:88)
org.elasticsearch.index.query.MultiMatchQueryParser.parse(MultiMatchQueryParser.java:164)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:107)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.functionscore.FunctionScoreQueryParser.parse(FunctionScoreQueryParser.java:104)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:71)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
23.9% (119.3ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#10]'
2/10 snapshots sharing following 9 elements
org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
7/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
unique snapshot
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
java.lang.ThreadLocal.get(ThreadLocal.java:163)
org.apache.lucene.util.CloseableThreadLocal.get(CloseableThreadLocal.java:78)
org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:164)
org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:586)
org.apache.lucene.search.similarities.TFIDFSimilarity.simScorer(TFIDFSimilarity.java:698)
org.apache.lucene.search.similarities.PerFieldSimilarityWrapper.simScorer(PerFieldSimilarityWrapper.java:59)
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:86)
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.scorer(DisjunctionMaxQuery.java:160)
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.scorer(FunctionScoreQuery.java:128)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.elasticsearch.common.lucene.search.XFilteredQuery$CustomRandomAccessFilterStrategy.filteredScorer(XFilteredQuery.java:233)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
21.6% (107.8ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#1]'
2/10 snapshots sharing following 3 elements
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
::: [xx-host-1100][GqUxLTglTSiUqStHp62DEg][xx-host-1100.domain.com][inet[xx-host-1100.domain.com/1100_ip:9300]]{aws_availability_zone=us-east-1a, master=true}
Hot threads at 2016-01-12T20:51:55.963Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
51.8% (259ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#5]'
2/10 snapshots sharing following 16 elements
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:150)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
8/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
48.6% (243.1ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#12]'
3/10 snapshots sharing following 34 elements
java.lang.System.identityHashCode(Native Method)
org.apache.lucene.util.WeakIdentityMap$IdentityWeakReference.<init>(WeakIdentityMap.java:250)
org.apache.lucene.util.WeakIdentityMap.put(WeakIdentityMap.java:131)
org.apache.lucene.store.ByteBufferIndexInput.buildSlice(ByteBufferIndexInput.java:296)
org.apache.lucene.store.ByteBufferIndexInput.clone(ByteBufferIndexInput.java:259)
org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl.clone(ByteBufferIndexInput.java:411)
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.initIndexInput(SegmentTermsEnum.java:117)
org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:143)
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:506)
org.apache.lucene.index.TermContext.build(TermContext.java:95)
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:165)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.<init>(DisjunctionMaxQuery.java:123)
org.apache.lucene.search.DisjunctionMaxQuery.createWeight(DisjunctionMaxQuery.java:201)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery.createWeight(FunctionScoreQuery.java:95)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:684)
org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:133)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 34 elements
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
java.lang.ThreadLocal.get(ThreadLocal.java:163)
org.elasticsearch.index.query.QueryParseContext.getTypes(QueryParseContext.java:79)
org.elasticsearch.index.query.QueryParseContext.smartFieldMappers(QueryParseContext.java:402)
org.elasticsearch.index.search.MatchQuery.parse(MatchQuery.java:170)
org.elasticsearch.index.search.MultiMatchQuery.parseAndApply(MultiMatchQuery.java:57)
org.elasticsearch.index.search.MultiMatchQuery.access$000(MultiMatchQuery.java:44)
org.elasticsearch.index.search.MultiMatchQuery$QueryBuilder.parseGroup(MultiMatchQuery.java:120)
org.elasticsearch.index.search.MultiMatchQuery$QueryBuilder.buildGroupedQueries(MultiMatchQuery.java:111)
org.elasticsearch.index.search.MultiMatchQuery.parse(MultiMatchQuery.java:88)
org.elasticsearch.index.query.MultiMatchQueryParser.parse(MultiMatchQueryParser.java:164)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:107)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.functionscore.FunctionScoreQueryParser.parse(FunctionScoreQueryParser.java:104)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:71)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
5/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
48.3% (241.5ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#4]'
2/10 snapshots sharing following 19 elements
org.elasticsearch.common.lucene.search.XFilteredQuery$CustomRandomAccessFilterStrategy.filteredScorer(XFilteredQuery.java:233)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 17 elements
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:150)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 11 elements
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:150)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 31 elements
org.apache.lucene.store.ByteBufferIndexInput.buildSlice(ByteBufferIndexInput.java:296)
org.apache.lucene.store.ByteBufferIndexInput.clone(ByteBufferIndexInput.java:259)
org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl.clone(ByteBufferIndexInput.java:411)
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.initIndexInput(SegmentTermsEnum.java:117)
org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:143)
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:506)
org.apache.lucene.index.TermContext.build(TermContext.java:95)
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:165)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.<init>(DisjunctionMaxQuery.java:123)
org.apache.lucene.search.DisjunctionMaxQuery.createWeight(DisjunctionMaxQuery.java:201)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery.createWeight(FunctionScoreQuery.java:95)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:684)
org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:133)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Raw
jan-12-2.txt
::: [xx-host-1400][Y4qXPFeeRK2uKbggdiReig][xx-host-1400.domain.com][inet[/1400_ip:9300]]{aws_availability_zone=us-east-1d, master=true}
Hot threads at 2016-01-12T20:52:09.800Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
31.4% (157ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#7]'
3/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
6/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
31.2% (155.9ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#10]'
2/10 snapshots sharing following 15 elements
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
8/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
30.4% (151.8ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#11]'
2/10 snapshots sharing following 11 elements
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:335)
org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:427)
org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:219)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:184)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:401)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 3 elements
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
::: [xx-host-1200][i0oiyUh4Tb-KtDUkOFzzXg][xx-host-1200.domain.com][inet[/1200_ip:9300]]{aws_availability_zone=us-east-1b, master=true}
Hot threads at 2016-01-12T20:52:09.798Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
36.8% (184.1ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#4]'
2/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
6/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
30.0% (150.1ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#1]'
3/10 snapshots sharing following 10 elements
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
6/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
::: [xx-host-1100][GqUxLTglTSiUqStHp62DEg][xx-host-1100.domain.com][inet[xx-host-1100.domain.com/1100_ip:9300]]{aws_availability_zone=us-east-1a, master=true}
Hot threads at 2016-01-12T20:52:09.807Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
55.6% (278ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#9]'
2/10 snapshots sharing following 18 elements
org.apache.lucene.util.fst.FST.findTargetArc(FST.java:1271)
org.apache.lucene.util.fst.FST.findTargetArc(FST.java:1195)
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:482)
org.elasticsearch.common.lucene.index.FilterableTermsEnum.seekExact(FilterableTermsEnum.java:131)
org.elasticsearch.search.aggregations.bucket.significant.SignificantTermsAggregatorFactory.getBackgroundFrequency(SignificantTermsAggregatorFactory.java:250)
org.elasticsearch.search.aggregations.bucket.significant.SignificantTermsAggregatorFactory.getBackgroundFrequency(SignificantTermsAggregatorFactory.java:262)
org.elasticsearch.search.aggregations.bucket.significant.SignificantLongTermsAggregator.buildAggregation(SignificantLongTermsAggregator.java:77)
org.elasticsearch.search.aggregations.bucket.significant.SignificantLongTermsAggregator.buildAggregation(SignificantLongTermsAggregator.java:40)
org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:134)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:172)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:301)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:312)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 16 elements
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:506)
org.elasticsearch.common.lucene.index.FilterableTermsEnum.seekExact(FilterableTermsEnum.java:131)
org.elasticsearch.search.aggregations.bucket.significant.SignificantTermsAggregatorFactory.getBackgroundFrequency(SignificantTermsAggregatorFactory.java:250)
org.elasticsearch.search.aggregations.bucket.significant.SignificantTermsAggregatorFactory.getBackgroundFrequency(SignificantTermsAggregatorFactory.java:262)
org.elasticsearch.search.aggregations.bucket.significant.SignificantLongTermsAggregator.buildAggregation(SignificantLongTermsAggregator.java:77)
org.elasticsearch.search.aggregations.bucket.significant.SignificantLongTermsAggregator.buildAggregation(SignificantLongTermsAggregator.java:40)
org.elasticsearch.search.aggregations.AggregationPhase.execute(AggregationPhase.java:134)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:172)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:301)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:312)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 10 elements
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
52.3% (261.3ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#5]'
4/10 snapshots sharing following 17 elements
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 34 elements
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
java.lang.ThreadLocal.get(ThreadLocal.java:163)
org.elasticsearch.index.query.QueryParseContext.getTypes(QueryParseContext.java:79)
org.elasticsearch.index.query.QueryParseContext.smartFieldMappers(QueryParseContext.java:402)
org.elasticsearch.index.search.MatchQuery.parse(MatchQuery.java:170)
org.elasticsearch.index.search.MultiMatchQuery.parseAndApply(MultiMatchQuery.java:57)
org.elasticsearch.index.search.MultiMatchQuery.access$000(MultiMatchQuery.java:44)
org.elasticsearch.index.search.MultiMatchQuery$QueryBuilder.parseGroup(MultiMatchQuery.java:120)
org.elasticsearch.index.search.MultiMatchQuery$QueryBuilder.buildGroupedQueries(MultiMatchQuery.java:111)
org.elasticsearch.index.search.MultiMatchQuery.parse(MultiMatchQuery.java:88)
org.elasticsearch.index.query.MultiMatchQueryParser.parse(MultiMatchQueryParser.java:164)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:107)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.functionscore.FunctionScoreQueryParser.parse(FunctionScoreQueryParser.java:104)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:71)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
51.1% (255.2ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#7]'
2/10 snapshots sharing following 19 elements
org.elasticsearch.common.lucene.search.XFilteredQuery$CustomRandomAccessFilterStrategy.filteredScorer(XFilteredQuery.java:233)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 15 elements
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 24 elements
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:165)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.<init>(DisjunctionMaxQuery.java:123)
org.apache.lucene.search.DisjunctionMaxQuery.createWeight(DisjunctionMaxQuery.java:201)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery.createWeight(FunctionScoreQuery.java:95)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:684)
org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:133)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Raw
jan-12-3.txt
::: [xx-host-1400][Y4qXPFeeRK2uKbggdiReig][xx-host-1400.domain.com][inet[/1400_ip:9300]]{aws_availability_zone=us-east-1d, master=true}
Hot threads at 2016-01-12T20:52:15.416Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
37.0% (185ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#2]'
5/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
5/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
36.6% (183.1ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#1]'
2/10 snapshots sharing following 10 elements
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
unique snapshot
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
29.8% (148.8ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#5]'
5/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
::: [xx-host-1200][i0oiyUh4Tb-KtDUkOFzzXg][xx-host-1200.domain.com][inet[/1200_ip:9300]]{aws_availability_zone=us-east-1b, master=true}
Hot threads at 2016-01-12T20:52:15.415Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
21.6% (107.8ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#2]'
2/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
21.5% (107.3ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#6]'
2/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
20.2% (101.1ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#4]'
unique snapshot
java.lang.System.identityHashCode(Native Method)
org.apache.lucene.util.WeakIdentityMap$IdentityWeakReference.<init>(WeakIdentityMap.java:250)
org.apache.lucene.util.WeakIdentityMap.put(WeakIdentityMap.java:131)
org.apache.lucene.store.ByteBufferIndexInput.buildSlice(ByteBufferIndexInput.java:296)
org.apache.lucene.store.ByteBufferIndexInput.clone(ByteBufferIndexInput.java:259)
org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl.clone(ByteBufferIndexInput.java:411)
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.initIndexInput(SegmentTermsEnum.java:117)
org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:143)
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:506)
org.apache.lucene.index.TermContext.build(TermContext.java:95)
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:165)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.<init>(DisjunctionMaxQuery.java:123)
org.apache.lucene.search.DisjunctionMaxQuery.createWeight(DisjunctionMaxQuery.java:201)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery.createWeight(FunctionScoreQuery.java:95)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:684)
org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:133)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
::: [xx-host-1100][GqUxLTglTSiUqStHp62DEg][xx-host-1100.domain.com][inet[xx-host-1100.domain.com/1100_ip:9300]]{aws_availability_zone=us-east-1a, master=true}
Hot threads at 2016-01-12T20:52:15.415Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
42.4% (211.8ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#4]'
2/10 snapshots sharing following 13 elements
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 3 elements
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
41.4% (206.8ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#13]'
2/10 snapshots sharing following 8 elements
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 10 elements
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
40.9% (204.3ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#9]'
2/10 snapshots sharing following 10 elements
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
6/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Raw
jan-12-4.txt
::: [xx-host-1400][Y4qXPFeeRK2uKbggdiReig][xx-host-1400.domain.com][inet[/1400_ip:9300]]{aws_availability_zone=us-east-1d, master=true}
Hot threads at 2016-01-12T20:52:20.511Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
34.0% (170.1ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#5]'
2/10 snapshots sharing following 25 elements
org.apache.lucene.index.TermContext.build(TermContext.java:95)
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:165)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.<init>(DisjunctionMaxQuery.java:123)
org.apache.lucene.search.DisjunctionMaxQuery.createWeight(DisjunctionMaxQuery.java:201)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery.createWeight(FunctionScoreQuery.java:95)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:684)
org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:133)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 34 elements
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
java.lang.ThreadLocal.get(ThreadLocal.java:163)
org.apache.lucene.util.CloseableThreadLocal.get(CloseableThreadLocal.java:78)
org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:164)
org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:586)
org.apache.lucene.search.similarities.TFIDFSimilarity.simScorer(TFIDFSimilarity.java:698)
org.apache.lucene.search.similarities.PerFieldSimilarityWrapper.simScorer(PerFieldSimilarityWrapper.java:59)
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:86)
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.scorer(DisjunctionMaxQuery.java:160)
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.scorer(FunctionScoreQuery.java:128)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.elasticsearch.common.lucene.search.XFilteredQuery$CustomRandomAccessFilterStrategy.filteredScorer(XFilteredQuery.java:233)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 8 elements
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 17 elements
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
32.9% (164.3ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#4]'
2/10 snapshots sharing following 21 elements
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:736)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:726)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:90)
org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:135)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:314)
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:335)
org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:427)
org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:219)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:184)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:401)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
6/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
32.1% (160.3ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#10]'
8/10 snapshots sharing following 3 elements
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
::: [xx-host-1200][i0oiyUh4Tb-KtDUkOFzzXg][xx-host-1200.domain.com][inet[/1200_ip:9300]]{aws_availability_zone=us-east-1b, master=true}
Hot threads at 2016-01-12T20:52:20.531Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
50.3% (251.3ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#7]'
6/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
33.7% (168.2ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#3]'
8/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
33.5% (167.4ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#8]'
3/10 snapshots sharing following 10 elements
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
unique snapshot
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
::: [xx-host-1100][GqUxLTglTSiUqStHp62DEg][xx-host-1100.domain.com][inet[xx-host-1100.domain.com/1100_ip:9300]]{aws_availability_zone=us-east-1a, master=true}
Hot threads at 2016-01-12T20:52:20.511Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
55.0% (275.2ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#4]'
3/10 snapshots sharing following 10 elements
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 19 elements
org.apache.lucene.search.FieldComparator$FloatComparator.compareBottom(FieldComparator.java:440)
org.apache.lucene.search.TopFieldCollector$OutOfOrderOneComparatorNonScoringCollector.collect(TopFieldCollector.java:126)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:581)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:150)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 15 elements
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
unique snapshot
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:397)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
53.5% (267.7ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#10]'
2/10 snapshots sharing following 25 elements
org.apache.lucene.index.TermContext.build(TermContext.java:95)
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:165)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.<init>(DisjunctionMaxQuery.java:123)
org.apache.lucene.search.DisjunctionMaxQuery.createWeight(DisjunctionMaxQuery.java:201)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery.createWeight(FunctionScoreQuery.java:95)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:684)
org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:133)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 7 elements
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 34 elements
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
java.lang.ThreadLocal.get(ThreadLocal.java:163)
org.elasticsearch.index.query.QueryParseContext.getTypes(QueryParseContext.java:79)
org.elasticsearch.index.query.QueryParseContext.smartFieldMappers(QueryParseContext.java:402)
org.elasticsearch.index.search.MatchQuery.parse(MatchQuery.java:170)
org.elasticsearch.index.search.MultiMatchQuery.parseAndApply(MultiMatchQuery.java:57)
org.elasticsearch.index.search.MultiMatchQuery.access$000(MultiMatchQuery.java:44)
org.elasticsearch.index.search.MultiMatchQuery$QueryBuilder.parseGroup(MultiMatchQuery.java:120)
org.elasticsearch.index.search.MultiMatchQuery$QueryBuilder.buildGroupedQueries(MultiMatchQuery.java:111)
org.elasticsearch.index.search.MultiMatchQuery.parse(MultiMatchQuery.java:88)
org.elasticsearch.index.query.MultiMatchQueryParser.parse(MultiMatchQueryParser.java:164)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:107)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.functionscore.FunctionScoreQueryParser.parse(FunctionScoreQueryParser.java:104)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:71)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 22 elements
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:736)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:726)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:101)
org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:135)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:314)
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:335)
org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:427)
org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:219)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:184)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:401)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
52.8% (263.9ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#8]'
7/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Raw
jan-12-5.txt
::: [xx-host-1400][Y4qXPFeeRK2uKbggdiReig][xx-host-1400.domain.com][inet[/1400_ip:9300]]{aws_availability_zone=us-east-1d, master=true}
Hot threads at 2016-01-12T20:52:27.759Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
33.0% (164.8ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#9]'
2/10 snapshots sharing following 34 elements
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
java.lang.ThreadLocal.get(ThreadLocal.java:163)
org.apache.lucene.util.CloseableThreadLocal.get(CloseableThreadLocal.java:78)
org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:164)
org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:586)
org.apache.lucene.search.similarities.TFIDFSimilarity.simScorer(TFIDFSimilarity.java:698)
org.apache.lucene.search.similarities.PerFieldSimilarityWrapper.simScorer(PerFieldSimilarityWrapper.java:59)
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:86)
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.scorer(DisjunctionMaxQuery.java:160)
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.scorer(FunctionScoreQuery.java:128)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.elasticsearch.common.lucene.search.XFilteredQuery$CustomRandomAccessFilterStrategy.filteredScorer(XFilteredQuery.java:233)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
29.0% (145ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#5]'
2/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
7/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
28.5% (142.5ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#7]'
2/10 snapshots sharing following 14 elements
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:684)
org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:133)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 34 elements
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
java.lang.ThreadLocal.get(ThreadLocal.java:163)
org.apache.lucene.util.CloseableThreadLocal.get(CloseableThreadLocal.java:78)
org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:164)
org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:586)
org.apache.lucene.search.similarities.TFIDFSimilarity.simScorer(TFIDFSimilarity.java:698)
org.apache.lucene.search.similarities.PerFieldSimilarityWrapper.simScorer(PerFieldSimilarityWrapper.java:59)
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:86)
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.scorer(DisjunctionMaxQuery.java:160)
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.scorer(FunctionScoreQuery.java:128)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.elasticsearch.common.lucene.search.XFilteredQuery$CustomRandomAccessFilterStrategy.filteredScorer(XFilteredQuery.java:233)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
unique snapshot
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
java.lang.ThreadLocal.get(ThreadLocal.java:163)
org.elasticsearch.index.query.QueryParseContext.getTypes(QueryParseContext.java:79)
org.elasticsearch.index.query.QueryParseContext.smartFieldMappers(QueryParseContext.java:402)
org.elasticsearch.index.search.MatchQuery.parse(MatchQuery.java:170)
org.elasticsearch.index.search.MultiMatchQuery.parseAndApply(MultiMatchQuery.java:57)
org.elasticsearch.index.search.MultiMatchQuery.access$000(MultiMatchQuery.java:44)
org.elasticsearch.index.search.MultiMatchQuery$QueryBuilder.parseGroup(MultiMatchQuery.java:120)
org.elasticsearch.index.search.MultiMatchQuery$QueryBuilder.buildGroupedQueries(MultiMatchQuery.java:111)
org.elasticsearch.index.search.MultiMatchQuery.parse(MultiMatchQuery.java:88)
org.elasticsearch.index.query.MultiMatchQueryParser.parse(MultiMatchQueryParser.java:164)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:107)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.functionscore.FunctionScoreQueryParser.parse(FunctionScoreQueryParser.java:104)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:71)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
::: [xx-host-1200][i0oiyUh4Tb-KtDUkOFzzXg][xx-host-1200.domain.com][inet[/1200_ip:9300]]{aws_availability_zone=us-east-1b, master=true}
Hot threads at 2016-01-12T20:52:27.758Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
29.2% (146.1ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#3]'
2/10 snapshots sharing following 17 elements
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 8 elements
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 10 elements
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
25.4% (127ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#4]'
8/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
24.3% (121.2ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#12]'
2/10 snapshots sharing following 17 elements
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
5/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
::: [xx-host-1100][GqUxLTglTSiUqStHp62DEg][xx-host-1100.domain.com][inet[xx-host-1100.domain.com/1100_ip:9300]]{aws_availability_zone=us-east-1a, master=true}
Hot threads at 2016-01-12T20:52:27.759Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
48.6% (243.2ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#13]'
2/10 snapshots sharing following 10 elements
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
8/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
47.6% (237.8ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#5]'
3/10 snapshots sharing following 22 elements
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.nextDoc(Lucene41PostingsReader.java:446)
org.apache.lucene.search.TermScorer.nextDoc(TermScorer.java:65)
org.apache.lucene.search.DisjunctionScorer.nextDoc(DisjunctionScorer.java:135)
org.elasticsearch.common.lucene.search.function.CustomBoostFactorScorer$AnyNextDoc.nextDoc(CustomBoostFactorScorer.java:128)
org.elasticsearch.common.lucene.search.function.CustomBoostFactorScorer.nextDoc(CustomBoostFactorScorer.java:64)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:192)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 13 elements
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 4 elements
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
46.2% (231ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#9]'
3/10 snapshots sharing following 34 elements
java.lang.System.identityHashCode(Native Method)
org.apache.lucene.util.WeakIdentityMap$IdentityWeakReference.<init>(WeakIdentityMap.java:250)
org.apache.lucene.util.WeakIdentityMap.put(WeakIdentityMap.java:131)
org.apache.lucene.store.ByteBufferIndexInput.buildSlice(ByteBufferIndexInput.java:296)
org.apache.lucene.store.ByteBufferIndexInput.clone(ByteBufferIndexInput.java:259)
org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl.clone(ByteBufferIndexInput.java:411)
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.initIndexInput(SegmentTermsEnum.java:117)
org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:143)
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:506)
org.apache.lucene.index.TermContext.build(TermContext.java:95)
org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:165)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.<init>(DisjunctionMaxQuery.java:123)
org.apache.lucene.search.DisjunctionMaxQuery.createWeight(DisjunctionMaxQuery.java:201)
org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:185)
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:526)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery.createWeight(FunctionScoreQuery.java:95)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.FilteredQuery.createWeight(FilteredQuery.java:82)
org.elasticsearch.common.lucene.search.XFilteredQuery.createWeight(XFilteredQuery.java:86)
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:684)
org.elasticsearch.search.internal.ContextIndexSearcher.createNormalizedWeight(ContextIndexSearcher.java:133)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
6/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Raw
jan-12-6.txt
::: [xx-host-1400][Y4qXPFeeRK2uKbggdiReig][xx-host-1400.domain.com][inet[/1400_ip:9300]]{aws_availability_zone=us-east-1d, master=true}
Hot threads at 2016-01-12T20:52:33.984Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
36.5% (182.5ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#10]'
3/10 snapshots sharing following 11 elements
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:150)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 15 elements
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
unique snapshot
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
java.lang.ThreadLocal.get(ThreadLocal.java:163)
org.apache.lucene.util.CloseableThreadLocal.get(CloseableThreadLocal.java:78)
org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:164)
org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:586)
org.apache.lucene.search.similarities.TFIDFSimilarity.simScorer(TFIDFSimilarity.java:698)
org.apache.lucene.search.similarities.PerFieldSimilarityWrapper.simScorer(PerFieldSimilarityWrapper.java:59)
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:86)
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.scorer(DisjunctionMaxQuery.java:160)
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.scorer(FunctionScoreQuery.java:128)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.elasticsearch.common.lucene.search.XFilteredQuery$CustomRandomAccessFilterStrategy.filteredScorer(XFilteredQuery.java:233)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
35.9% (179.5ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#13]'
2/10 snapshots sharing following 10 elements
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
35.3% (176.5ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1400][search][T#1]'
2/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
::: [xx-host-1200][i0oiyUh4Tb-KtDUkOFzzXg][xx-host-1200.domain.com][inet[/1200_ip:9300]]{aws_availability_zone=us-east-1b, master=true}
Hot threads at 2016-01-12T20:52:33.983Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
30.4% (152.1ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#6]'
2/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
24.6% (122.7ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#13]'
2/10 snapshots sharing following 8 elements
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
7/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
23.2% (115.7ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1200][search][T#2]'
2/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
7/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
::: [xx-host-1100][GqUxLTglTSiUqStHp62DEg][xx-host-1100.domain.com][inet[xx-host-1100.domain.com/1100_ip:9300]]{aws_availability_zone=us-east-1a, master=true}
Hot threads at 2016-01-12T20:52:33.983Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
53.9% (269.7ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#1]'
3/10 snapshots sharing following 34 elements
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
java.lang.ThreadLocal.get(ThreadLocal.java:163)
org.apache.lucene.util.CloseableThreadLocal.get(CloseableThreadLocal.java:78)
org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:164)
org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:586)
org.apache.lucene.search.similarities.TFIDFSimilarity.simScorer(TFIDFSimilarity.java:698)
org.apache.lucene.search.similarities.PerFieldSimilarityWrapper.simScorer(PerFieldSimilarityWrapper.java:59)
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:86)
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.scorer(DisjunctionMaxQuery.java:160)
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.scorer(FunctionScoreQuery.java:128)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.elasticsearch.common.lucene.search.XFilteredQuery$CustomRandomAccessFilterStrategy.filteredScorer(XFilteredQuery.java:233)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 34 elements
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
java.lang.ThreadLocal.get(ThreadLocal.java:163)
org.elasticsearch.index.query.QueryParseContext.getTypes(QueryParseContext.java:79)
org.elasticsearch.index.query.QueryParseContext.smartFieldMappers(QueryParseContext.java:402)
org.elasticsearch.index.search.MatchQuery.parse(MatchQuery.java:170)
org.elasticsearch.index.search.MultiMatchQuery.parseAndApply(MultiMatchQuery.java:57)
org.elasticsearch.index.search.MultiMatchQuery.access$000(MultiMatchQuery.java:44)
org.elasticsearch.index.search.MultiMatchQuery$QueryBuilder.parseGroup(MultiMatchQuery.java:120)
org.elasticsearch.index.search.MultiMatchQuery$QueryBuilder.buildGroupedQueries(MultiMatchQuery.java:111)
org.elasticsearch.index.search.MultiMatchQuery.parse(MultiMatchQuery.java:88)
org.elasticsearch.index.query.MultiMatchQueryParser.parse(MultiMatchQueryParser.java:164)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:107)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.functionscore.FunctionScoreQueryParser.parse(FunctionScoreQueryParser.java:104)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:71)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 6 elements
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
42.3% (211.5ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#7]'
2/10 snapshots sharing following 8 elements
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
4/10 snapshots sharing following 19 elements
org.elasticsearch.common.lucene.search.XFilteredQuery$CustomRandomAccessFilterStrategy.filteredScorer(XFilteredQuery.java:233)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:136)
org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542)
org.apache.lucene.search.FilteredQuery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504)
org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
unique snapshot
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:736)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:726)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:101)
org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:135)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:354)
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:335)
org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:427)
org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:219)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:184)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:401)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
40.1% (200.6ms out of 500ms) cpu usage by thread 'elasticsearch[xx-host-1100][search][T#6]'
2/10 snapshots sharing following 10 elements
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:391)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 18 elements
org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
org.elasticsearch.common.netty.channel.Channels.write(Channels.java:725)
org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87)
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:195)
org.elasticsearch.rest.action.support.RestResponseListener.processResponse(RestResponseListener.java:43)
org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49)
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction$1.doRun(TransportSearchQueryAndFetchAction.java:88)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 24 elements
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
java.lang.ThreadLocal.get(ThreadLocal.java:163)
org.elasticsearch.search.internal.SearchContext.current(SearchContext.java:96)
org.elasticsearch.index.query.functionscore.fieldvaluefactor.FieldValueFactorFunctionParser.parse(FieldValueFactorFunctionParser.java:86)
org.elasticsearch.index.query.functionscore.FunctionScoreQueryParser.parse(FunctionScoreQueryParser.java:133)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:71)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
unique snapshot
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
java.lang.ThreadLocal.get(ThreadLocal.java:163)
org.elasticsearch.index.query.QueryParseContext.getTypes(QueryParseContext.java:79)
org.elasticsearch.index.query.QueryParseContext.smartFieldMappers(QueryParseContext.java:402)
org.elasticsearch.index.query.RangeFilterParser.parse(RangeFilterParser.java:133)
org.elasticsearch.index.query.QueryParseContext.executeFilterParser(QueryParseContext.java:373)
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:354)
org.elasticsearch.index.query.BoolFilterParser.parse(BoolFilterParser.java:92)
org.elasticsearch.index.query.QueryParseContext.executeFilterParser(QueryParseContext.java:373)
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:354)
org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:74)
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:307)
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:385)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment