Skip to content

Instantly share code, notes, and snippets.

@robinjoseph08

robinjoseph08/hot_threads.txt Secret

Created July 17, 2018 21:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save robinjoseph08/0cd5ea281a33eb18ea4425608ed3e38f to your computer and use it in GitHub Desktop.
Save robinjoseph08/0cd5ea281a33eb18ea4425608ed3e38f to your computer and use it in GitHub Desktop.
Download ZIP
Elasticsearch Hot Threads
Raw
hot_threads.txt
::: {ip-10-200-128-140}{yW7s_UVTREa1zx3dXtGWTw}{lXS-fDKgTm6c59AL8fUHKA}{10.200.128.140}{10.200.128.140:9300}{aws_availability_zone=us-west-2a, ml.machine_memory=32665927680, ml.max_open_jobs=20, ml.enabled=true}
Hot threads at 2018-07-17T21:38:36.516Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.9% (504.3ms out of 500ms) cpu usage by thread 'elasticsearch[ip-10-200-128-140][search][T#11]'
10/10 snapshots sharing following 23 elements
org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregator$1.collect(DateHistogramAggregator.java:108)
org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:82)
org.apache.lucene.search.MultiCollector$MultiLeafCollector.collect(MultiCollector.java:174)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:233)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:184)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:660)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:265)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:307)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:340)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:316)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:312)
org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1002)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
100.9% (504.3ms out of 500ms) cpu usage by thread 'elasticsearch[ip-10-200-128-140][search][T#15]'
2/10 snapshots sharing following 29 elements
org.joda.time.tz.DateTimeZoneBuilder$OfYear.previous(DateTimeZoneBuilder.java:609)
org.joda.time.tz.DateTimeZoneBuilder$Recurrence.previous(DateTimeZoneBuilder.java:772)
org.joda.time.tz.DateTimeZoneBuilder$DSTZone.previousTransition(DateTimeZoneBuilder.java:1297)
org.joda.time.tz.DateTimeZoneBuilder$PrecalculatedZone.previousTransition(DateTimeZoneBuilder.java:1638)
org.joda.time.tz.CachedDateTimeZone.previousTransition(CachedDateTimeZone.java:119)
org.elasticsearch.common.rounding.Rounding$TimeIntervalRounding.round(Rounding.java:247)
org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregator$1.collect(DateHistogramAggregator.java:108)
org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:82)
org.apache.lucene.search.MultiCollector$MultiLeafCollector.collect(MultiCollector.java:174)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:233)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:184)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:660)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:265)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:307)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:340)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:316)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:312)
org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1002)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
8/10 snapshots sharing following 21 elements
org.apache.lucene.search.MultiCollector$MultiLeafCollector.collect(MultiCollector.java:174)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:233)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:184)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:660)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:265)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:307)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:340)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:316)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:312)
org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1002)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
100.9% (504.2ms out of 500ms) cpu usage by thread 'elasticsearch[ip-10-200-128-140][search][T#12]'
2/10 snapshots sharing following 29 elements
org.joda.time.tz.DateTimeZoneBuilder$OfYear.previous(DateTimeZoneBuilder.java:624)
org.joda.time.tz.DateTimeZoneBuilder$Recurrence.previous(DateTimeZoneBuilder.java:772)
org.joda.time.tz.DateTimeZoneBuilder$DSTZone.previousTransition(DateTimeZoneBuilder.java:1297)
org.joda.time.tz.DateTimeZoneBuilder$PrecalculatedZone.previousTransition(DateTimeZoneBuilder.java:1638)
org.joda.time.tz.CachedDateTimeZone.previousTransition(CachedDateTimeZone.java:119)
org.elasticsearch.common.rounding.Rounding$TimeIntervalRounding.round(Rounding.java:247)
org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregator$1.collect(DateHistogramAggregator.java:108)
org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:82)
org.apache.lucene.search.MultiCollector$MultiLeafCollector.collect(MultiCollector.java:174)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:233)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:184)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:660)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:265)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:307)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:340)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:316)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:312)
org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1002)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
3/10 snapshots sharing following 22 elements
org.apache.lucene.search.MultiCollector$MultiLeafCollector.collect(MultiCollector.java:174)
org.apache.lucene.search.MultiCollector$MultiLeafCollector.collect(MultiCollector.java:174)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:233)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:184)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:660)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:265)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:307)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:340)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:316)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:312)
org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1002)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
5/10 snapshots sharing following 25 elements
org.joda.time.tz.CachedDateTimeZone.previousTransition(CachedDateTimeZone.java:119)
org.elasticsearch.common.rounding.Rounding$TimeIntervalRounding.round(Rounding.java:247)
org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregator$1.collect(DateHistogramAggregator.java:108)
org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:82)
org.apache.lucene.search.MultiCollector$MultiLeafCollector.collect(MultiCollector.java:174)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:233)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:184)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:660)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:265)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:107)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:307)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:340)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:316)
org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:312)
org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1002)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
::: {ip-10-200-129-75}{uTVAbssnTS2voCY_gur1lQ}{KhhGlJWbQaawg6oyjYuQRg}{10.200.129.75}{10.200.129.75:9300}{aws_availability_zone=us-west-2b, ml.machine_memory=32665927680, ml.max_open_jobs=20, ml.enabled=true}
Hot threads at 2018-07-17T21:38:36.520Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {ip-10-200-130-191}{sBNjDNK4SDWVShWb9_M8mg}{Ll3_L3MTQeuE7J9AGxH2ZA}{10.200.130.191}{10.200.130.191:9300}{aws_availability_zone=us-west-2c, ml.machine_memory=32665927680, ml.max_open_jobs=20, ml.enabled=true}
Hot threads at 2018-07-17T21:38:36.521Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment