Skip to content

Instantly share code, notes, and snippets.

@Morriaty-The-Murderer

Morriaty-The-Murderer/hot_threads

Created June 24, 2020 08:48
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Morriaty-The-Murderer/8c0a075f328ffcb640a0f91707975ff2 to your computer and use it in GitHub Desktop.
Save Morriaty-The-Murderer/8c0a075f328ffcb640a0f91707975ff2 to your computer and use it in GitHub Desktop.
Download ZIP
output of hot threads
Raw
hot_threads
::: {matchtest_es-node-172.16.0.222}{gx6bGizaTkOtl_CN51F-OQ}{q2M6rN-ER3ynjCubw5YMiQ}{172.16.0.222}{172.16.0.222:9300}{ml.machine_memory=16272351232, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2020-06-24T08:28:57.335, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
46.6% (233.2ms out of 500ms) cpu usage by thread 'elasticsearch[matchtest_es-node-172.16.0.222][search][T#1]'
2/10 snapshots sharing following 21 elements
org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:133)
org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.document(CompressingStoredFieldsReader.java:555)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.document(CompressingStoredFieldsReader.java:571)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:578)
org.apache.lucene.index.CodecReader.document(CodecReader.java:84)
org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:341)
org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:427)
org.elasticsearch.search.fetch.FetchPhase.getSearchFields(FetchPhase.java:226)
org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:201)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:152)
org.elasticsearch.search.SearchService.lambda$executeFetchPhase$3(SearchService.java:556)
org.elasticsearch.search.SearchService$$Lambda$3662/1936852274.get(Unknown Source)
org.elasticsearch.search.SearchService$3.doRun(SearchService.java:361)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 11 elements
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:152)
org.elasticsearch.search.SearchService.lambda$executeFetchPhase$3(SearchService.java:556)
org.elasticsearch.search.SearchService$$Lambda$3662/1936852274.get(Unknown Source)
org.elasticsearch.search.SearchService$3.doRun(SearchService.java:361)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
6/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
42.0% (209.8ms out of 500ms) cpu usage by thread 'elasticsearch[matchtest_es-node-172.16.0.222][search][T#8]'
2/10 snapshots sharing following 24 elements
org.apache.lucene.search.Boolean2ScorerSupplier.get(Boolean2ScorerSupplier.java:92)
org.apache.lucene.search.ConstantScoreQuery$1$1.get(ConstantScoreQuery.java:132)
org.apache.lucene.search.ConstantScoreQuery$1.scorer(ConstantScoreQuery.java:164)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.functionScorer(FunctionScoreQuery.java:252)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.scorer(FunctionScoreQuery.java:274)
org.apache.lucene.search.Weight.bulkScorer(Weight.java:177)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:667)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:471)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:330)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)
org.elasticsearch.search.SearchService.access$100(SearchService.java:121)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335)
org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
3/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
3/10 snapshots sharing following 21 elements
org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:133)
org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.document(CompressingStoredFieldsReader.java:555)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.document(CompressingStoredFieldsReader.java:571)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:578)
org.apache.lucene.index.CodecReader.document(CodecReader.java:84)
org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:341)
org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:427)
org.elasticsearch.search.fetch.FetchPhase.getSearchFields(FetchPhase.java:226)
org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:201)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:152)
org.elasticsearch.search.SearchService.lambda$executeFetchPhase$3(SearchService.java:556)
org.elasticsearch.search.SearchService$$Lambda$3662/1936852274.get(Unknown Source)
org.elasticsearch.search.SearchService$3.doRun(SearchService.java:361)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
41.1% (205.6ms out of 500ms) cpu usage by thread 'elasticsearch[matchtest_es-node-172.16.0.222][search][T#12]'
2/10 snapshots sharing following 32 elements
org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:523)
org.apache.lucene.document.LatLonPointDistanceQuery$1$1.get(LatLonPointDistanceQuery.java:164)
org.apache.lucene.search.IndexOrDocValuesQuery$1$1.get(IndexOrDocValuesQuery.java:157)
org.apache.lucene.search.Boolean2ScorerSupplier.req(Boolean2ScorerSupplier.java:145)
org.apache.lucene.search.Boolean2ScorerSupplier.get(Boolean2ScorerSupplier.java:92)
org.apache.lucene.search.Boolean2ScorerSupplier.req(Boolean2ScorerSupplier.java:122)
org.apache.lucene.search.Boolean2ScorerSupplier.get(Boolean2ScorerSupplier.java:92)
org.apache.lucene.search.Boolean2ScorerSupplier.req(Boolean2ScorerSupplier.java:145)
org.apache.lucene.search.Boolean2ScorerSupplier.get(Boolean2ScorerSupplier.java:92)
org.apache.lucene.search.ConstantScoreQuery$1$1.get(ConstantScoreQuery.java:132)
org.apache.lucene.search.ConstantScoreQuery$1.scorer(ConstantScoreQuery.java:164)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.functionScorer(FunctionScoreQuery.java:252)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.scorer(FunctionScoreQuery.java:274)
org.apache.lucene.search.Weight.bulkScorer(Weight.java:177)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:667)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:471)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:330)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)
org.elasticsearch.search.SearchService.access$100(SearchService.java:121)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335)
org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
4/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 7 elements
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
::: {matchtest_es-node-172.16.0.223}{2rnoa1V2TZyb8hQC_nV3Nw}{nikll_llTo6M8rWa9WyOHA}{172.16.0.223}{172.16.0.223:9300}{ml.machine_memory=16272351232, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2020-06-24T08:28:57.337, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
69.9% (349.5ms out of 500ms) cpu usage by thread 'elasticsearch[matchtest_es-node-172.16.0.223][search][T#10]'
8/10 snapshots sharing following 21 elements
org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:133)
org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.document(CompressingStoredFieldsReader.java:555)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.document(CompressingStoredFieldsReader.java:571)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:578)
org.apache.lucene.index.CodecReader.document(CodecReader.java:84)
org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:341)
org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:427)
org.elasticsearch.search.fetch.FetchPhase.getSearchFields(FetchPhase.java:226)
org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:201)
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:152)
org.elasticsearch.search.SearchService.lambda$executeFetchPhase$3(SearchService.java:556)
org.elasticsearch.search.SearchService$$Lambda$3329/1879743341.get(Unknown Source)
org.elasticsearch.search.SearchService$3.doRun(SearchService.java:361)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 8 elements
org.elasticsearch.search.SearchService$3.doRun(SearchService.java:361)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
69.1% (345.4ms out of 500ms) cpu usage by thread 'elasticsearch[matchtest_es-node-172.16.0.223][search][T#6]'
2/10 snapshots sharing following 38 elements
org.apache.lucene.store.DataInput.readVLong(DataInput.java:184)
org.apache.lucene.store.DataInput.readVLong(DataInput.java:169)
org.apache.lucene.util.fst.FST.readUnpackedNodeTarget(FST.java:815)
org.apache.lucene.util.fst.FST.readNextRealArc(FST.java:1003)
org.apache.lucene.util.fst.FST.findTargetArc(FST.java:1117)
org.apache.lucene.util.fst.FST.findTargetArc(FST.java:1044)
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:483)
org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:145)
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:108)
org.apache.lucene.search.Weight.scorerSupplier(Weight.java:143)
org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:751)
org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:154)
org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:364)
org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:719)
org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:154)
org.apache.lucene.search.ConstantScoreQuery$1.scorerSupplier(ConstantScoreQuery.java:125)
org.apache.lucene.search.ConstantScoreQuery$1.scorer(ConstantScoreQuery.java:160)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.functionScorer(FunctionScoreQuery.java:252)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.scorer(FunctionScoreQuery.java:274)
org.apache.lucene.search.Weight.bulkScorer(Weight.java:177)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:667)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:471)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:330)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)
org.elasticsearch.search.SearchService.access$100(SearchService.java:121)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335)
org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 21 elements
org.elasticsearch.common.lucene.Lucene.writeTopDocs(Lucene.java:433)
org.elasticsearch.search.query.QuerySearchResult.writeToNoId(QuerySearchResult.java:329)
org.elasticsearch.search.query.QuerySearchResult.writeTo(QuerySearchResult.java:315)
org.elasticsearch.transport.TcpTransport.buildMessage(TcpTransport.java:1025)
org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:978)
org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:952)
org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:66)
org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:60)
org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)
org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onResponse(HandledTransportAction.java:103)
org.elasticsearch.action.support.HandledTransportAction$ChannelActionListener.onResponse(HandledTransportAction.java:87)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335)
org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 33 elements
org.apache.lucene.util.fst.FST.findTargetArc(FST.java:1044)
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:483)
org.apache.lucene.search.TermInSetQuery$1.rewrite(TermInSetQuery.java:256)
org.apache.lucene.search.TermInSetQuery$1.scorer(TermInSetQuery.java:322)
org.apache.lucene.search.Weight.scorerSupplier(Weight.java:143)
org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:719)
org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:154)
org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:364)
org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:719)
org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:154)
org.apache.lucene.search.ConstantScoreQuery$1.scorerSupplier(ConstantScoreQuery.java:125)
org.apache.lucene.search.ConstantScoreQuery$1.scorer(ConstantScoreQuery.java:160)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.functionScorer(FunctionScoreQuery.java:252)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.scorer(FunctionScoreQuery.java:274)
org.apache.lucene.search.Weight.bulkScorer(Weight.java:177)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:667)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:471)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:330)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)
org.elasticsearch.search.SearchService.access$100(SearchService.java:121)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335)
org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 17 elements
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:471)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:330)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)
org.elasticsearch.search.SearchService.access$100(SearchService.java:121)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335)
org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
67.5% (337.5ms out of 500ms) cpu usage by thread 'elasticsearch[matchtest_es-node-172.16.0.223][search][T#4]'
8/10 snapshots sharing following 10 elements
org.elasticsearch.search.SearchService.lambda$executeFetchPhase$3(SearchService.java:556)
org.elasticsearch.search.SearchService$$Lambda$3329/1879743341.get(Unknown Source)
org.elasticsearch.search.SearchService$3.doRun(SearchService.java:361)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 7 elements
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
::: {matchtest_es-node-172.16.0.221}{yAktZzaNTl2cq8lMQdBNCw}{7WoC2nXcStuGUE0fzW8geA}{172.16.0.221}{172.16.0.221:9300}{ml.machine_memory=16272351232, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
Hot threads at 2020-06-24T08:28:57.332, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
24.3% (121.3ms out of 500ms) cpu usage by thread 'elasticsearch[matchtest_es-node-172.16.0.221][search][T#12]'
2/10 snapshots sharing following 7 elements
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
6/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
22.0% (109.9ms out of 500ms) cpu usage by thread 'elasticsearch[matchtest_es-node-172.16.0.221][search][T#9]'
2/10 snapshots sharing following 29 elements
org.apache.lucene.search.Weight.scorerSupplier(Weight.java:143)
org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:725)
org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:154)
org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:364)
org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:719)
org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:154)
org.apache.lucene.search.ConstantScoreQuery$1.scorerSupplier(ConstantScoreQuery.java:125)
org.apache.lucene.search.ConstantScoreQuery$1.scorer(ConstantScoreQuery.java:160)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.functionScorer(FunctionScoreQuery.java:252)
org.elasticsearch.common.lucene.search.function.FunctionScoreQuery$CustomBoostFactorWeight.scorer(FunctionScoreQuery.java:274)
org.apache.lucene.search.Weight.bulkScorer(Weight.java:177)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:667)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:471)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:330)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)
org.elasticsearch.search.SearchService.access$100(SearchService.java:121)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335)
org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
6/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
20.2% (100.9ms out of 500ms) cpu usage by thread 'elasticsearch[matchtest_es-node-172.16.0.221][search][T#6]'
2/10 snapshots sharing following 7 elements
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
7/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment