Skip to content

Instantly share code, notes, and snippets.

@AmosIturan

AmosIturan/hot_threads.txt Secret

Created March 10, 2024 17:25
Show Gist options
  • Save AmosIturan/8d5fa73fdeb0f821616f5cc6a33e0a8a to your computer and use it in GitHub Desktop.
Save AmosIturan/8d5fa73fdeb0f821616f5cc6a33e0a8a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
hot_threads.txt
::: {eck-prod-es-master-nodes-0}{MY8JPCqqR8GutRdZw_DI5A}{hBY556CFS4SROsZ7s9NqeQ}{eck-prod-es-master-nodes-0}{10.233.109.173}{10.233.109.173:9300}{mr}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, k8s_node_name=elk8-manager-01}
Hot threads at 2024-03-10T16:33:50.982Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {eck-prod-es-master-nodes-1}{kMZspkTZQWGMAE4nhmVj7Q}{SXHdVnSgS7e61syE-2LiCw}{eck-prod-es-master-nodes-1}{10.233.72.108}{10.233.72.108:9300}{mr}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true, k8s_node_name=elk8-manager-02}
Hot threads at 2024-03-10T16:33:50.982Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {eck-prod-es-master-nodes-2}{pguI_6m2TPq0EZsyOI5cLQ}{0qkIzS2xQ62e9HVYxEPcvA}{eck-prod-es-master-nodes-2}{10.233.88.237}{10.233.88.237:9300}{mr}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true, k8s_node_name=elk8-manager-03}
Hot threads at 2024-03-10T16:33:50.984Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {eck-prod-es-data-cold-nodes-1}{7Vg3DTcYTvGLq2Y4AoMOnw}{6Eog6oJvRl-xM9nkT-2fkw}{eck-prod-es-data-cold-nodes-1}{10.233.78.234}{10.233.78.234:9300}{cf}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true, k8s_node_name=elk8-data-07}
Hot threads at 2024-03-10T16:33:50.982Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.5%, other=0.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-1][search_worker][T#14]'
9/10 snapshots sharing following 26 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fcfb1518838.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fcfb1518f78.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.FilterLeafReader.maxDoc(FilterLeafReader.java:392)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.FilterLeafReader.maxDoc(FilterLeafReader.java:392)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.FilterLeafReader.maxDoc(FilterLeafReader.java:392)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:385)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fcfb1518838.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fcfb1518f78.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.5%, other=0.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-1][search_worker][T#15]'
6/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fcfb1518838.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fcfb1518f78.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
4/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fcfb1518838.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fcfb1518f78.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.5%, other=0.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-1][search_worker][T#13]'
7/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fcfb1518838.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fcfb1518f78.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 28 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.get(BytesRefHash.java:129)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fcfb1518838.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fcfb1518f78.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.packed.DirectMonotonicReader.get(DirectMonotonicReader.java:153)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1121)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fcfb1518838.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fcfb1518f78.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-cold-nodes-3}{eFZokzgwRbKyGkS2SE50Vw}{IGCq4bqiS4iG5mZDq6Hr0Q}{eck-prod-es-data-cold-nodes-3}{10.233.82.43}{10.233.82.43:9300}{cf}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, k8s_node_name=elk8-data-11}
Hot threads at 2024-03-10T16:33:50.981Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.5%, other=0.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-3][search_worker][T#11]'
4/10 snapshots sharing following 27 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f322549dc30.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f322549e590.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
5/10 snapshots sharing following 26 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f322549dc30.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f322549e590.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f322549dc30.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f322549e590.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.5%, other=0.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-3][search_worker][T#9]'
7/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f322549dc30.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f322549e590.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 27 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f322549dc30.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f322549e590.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f322549dc30.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f322549e590.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.3%, other=0.7%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-3][search_worker][T#1]'
6/10 snapshots sharing following 25 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f322549dc30.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f322549e590.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
4/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f322549dc30.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f322549e590.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-cold-nodes-4}{gY0aOGO_T-q1JgqM_t5LwA}{YXPVlAkgRk-U7JsLcX0k1g}{eck-prod-es-data-cold-nodes-4}{10.233.120.170}{10.233.120.170:9300}{cf}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true, k8s_node_name=elk8-data-13}
Hot threads at 2024-03-10T16:33:50.982Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.5%, other=0.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-4][search_worker][T#13]'
7/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9929519728.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f992951a088.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 23 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9929519728.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f992951a088.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$21.ordValue(Lucene90DocValuesProducer.java:899)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.advanceExact(SingletonSortedSetDocValues.java:86)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.advanceExact(FieldData.java:303)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9929519728.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f992951a088.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.5%, other=0.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-4][search_worker][T#11]'
2/10 snapshots sharing following 29 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9929519728.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f992951a088.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
8/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9929519728.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f992951a088.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.5%, other=0.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-4][search_worker][T#14]'
3/10 snapshots sharing following 27 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9929519728.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f992951a088.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
4/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9929519728.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f992951a088.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9929519728.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f992951a088.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9929519728.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f992951a088.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-cold-nodes-0}{h7Bezl49Se-1v2Us1hRaGA}{_TJen4Y6TZCxN4zQBkLDRw}{eck-prod-es-data-cold-nodes-0}{10.233.100.172}{10.233.100.172:9300}{cf}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, k8s_node_name=elk8-data-08}
Hot threads at 2024-03-10T16:33:50.983Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.8%, other=0.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-0][search_worker][T#3]'
5/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f4d3549aa40.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f4d3549b180.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 28 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.get(BytesRefHash.java:129)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f4d3549aa40.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f4d3549b180.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f4d3549aa40.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f4d3549b180.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.6%, other=0.4%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-0][search_worker][T#15]'
3/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f4d3549aa40.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f4d3549b180.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
7/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f4d3549aa40.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f4d3549b180.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.5%, other=0.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-0][search_worker][T#7]'
4/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f4d3549aa40.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f4d3549b180.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
6/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f4d3549aa40.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f4d3549b180.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-cold-nodes-6}{zM_oxbBuRRKYBHOS4b_Bog}{FNyE0oMuQce82T6MsceWQg}{eck-prod-es-data-cold-nodes-6}{10.233.121.43}{10.233.121.43:9300}{cf}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true, k8s_node_name=elk8-data-16}
Hot threads at 2024-03-10T16:33:50.981Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.7%, other=0.3%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-6][search_worker][T#7]'
4/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f3a01465a90.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f3a014661d0.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 27 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f3a01465a90.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f3a014661d0.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f3a01465a90.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f3a014661d0.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f3a01465a90.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f3a014661d0.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.7%, other=0.3%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-6][search_worker][T#10]'
4/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:140)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f3a01465a90.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f3a014661d0.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.store.DataInput.readVInt(DataInput.java:121)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1248)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f3a01465a90.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f3a014661d0.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 27 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f3a01465a90.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f3a014661d0.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f3a01465a90.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f3a014661d0.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.7%, other=0.3%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-6][search_worker][T#4]'
7/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f3a01465a90.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f3a014661d0.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f3a01465a90.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f3a014661d0.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-cold-nodes-2}{8NGMDBolRdixdp-aKT5M-Q}{cL3KIF_sSYaJXMhoHmfNUA}{eck-prod-es-data-cold-nodes-2}{10.233.90.148}{10.233.90.148:9300}{cf}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true, k8s_node_name=elk8-data-18}
Hot threads at 2024-03-10T16:33:50.983Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.9%, other=0.1%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-2][search_worker][T#14]'
7/10 snapshots sharing following 28 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.get(BytesRefHash.java:129)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5594e2e58.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5594e3598.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:140)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5594e2e58.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5594e3598.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.8%, other=0.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-2][search_worker][T#6]'
5/10 snapshots sharing following 28 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.get(BytesRefHash.java:129)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5594e2e58.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5594e3598.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5594e2e58.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5594e3598.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5594e2e58.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5594e3598.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.8%, other=0.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-2][search_worker][T#5]'
2/10 snapshots sharing following 28 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.get(BytesRefHash.java:129)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5594e2e58.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5594e3598.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:140)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5594e2e58.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5594e3598.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5594e2e58.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5594e3598.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5594e2e58.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5594e3598.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-nodes-6}{VtPzX0MpREu3e1uBjvCfGQ}{0ETdZf7mS6Ol12YhlOLpig}{eck-prod-es-data-nodes-6}{10.233.89.107}{10.233.89.107:9300}{dhilrstw}{8.12.2}{7000099-8500010}{ml.allocated_processors_double=10.0, ml.machine_memory=64424509440, ml.config_version=12.0.0, ml.max_jvm_size=32212254720, ml.allocated_processors=10, xpack.installed=true, transform.config_version=10.0.0, k8s_node_name=elk8-data-05}
Hot threads at 2024-03-10T16:33:50.983Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.4%, other=0.6%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-6][search_worker][T#10]'
3/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:140)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fb1cd481e38.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fb1cd482578.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fb1cd481e38.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fb1cd482578.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 29 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefArray.get(BytesRefArray.java:107)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.get(BytesRefHash.java:129)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fb1cd481e38.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fb1cd482578.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 26 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.FilterLeafReader.maxDoc(FilterLeafReader.java:392)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.FilterLeafReader.maxDoc(FilterLeafReader.java:392)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.FilterLeafReader.maxDoc(FilterLeafReader.java:392)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:385)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fb1cd481e38.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fb1cd482578.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.3%, other=0.7%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-6][search_worker][T#9]'
3/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fb1cd481e38.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fb1cd482578.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
4/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fb1cd481e38.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fb1cd482578.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fb1cd481e38.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fb1cd482578.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.3%, other=0.7%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-6][search_worker][T#14]'
2/10 snapshots sharing following 29 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefArray.get(BytesRefArray.java:107)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.get(BytesRefHash.java:129)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fb1cd481e38.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fb1cd482578.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
6/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fb1cd481e38.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fb1cd482578.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 26 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.FilterLeafReader.maxDoc(FilterLeafReader.java:392)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.FilterLeafReader.maxDoc(FilterLeafReader.java:392)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.FilterLeafReader.maxDoc(FilterLeafReader.java:392)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:385)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fb1cd481e38.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fb1cd482578.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-nodes-8}{xP5F97S1R_iTsZht7znYjg}{osMbGHc0QsaSlFzjuHGrgw}{eck-prod-es-data-nodes-8}{10.233.108.43}{10.233.108.43:9300}{dhilrstw}{8.12.2}{7000099-8500010}{ml.allocated_processors=10, ml.max_jvm_size=32212254720, ml.config_version=12.0.0, ml.machine_memory=64424509440, ml.allocated_processors_double=10.0, transform.config_version=10.0.0, xpack.installed=true, k8s_node_name=elk8-data-12}
Hot threads at 2024-03-10T16:33:50.983Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.8%, other=0.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-8][search_worker][T#2]'
7/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe1314f51e8.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe1314f5928.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe1314f51e8.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe1314f5928.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.8%, other=0.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-8][search_worker][T#4]'
4/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:140)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe1314f51e8.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe1314f5928.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
4/10 snapshots sharing following 24 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe1314f51e8.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe1314f5928.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe1314f51e8.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe1314f5928.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.7%, other=0.3%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-8][search_worker][T#11]'
6/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe1314f51e8.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe1314f5928.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
4/10 snapshots sharing following 24 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe1314f51e8.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe1314f5928.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-nodes-9}{-j8SyxOISuyIegnwNAg8FQ}{sXW0XJFqQpyk7nIlUj308w}{eck-prod-es-data-nodes-9}{10.233.119.107}{10.233.119.107:9300}{dhilrstw}{8.12.2}{7000099-8500010}{ml.allocated_processors_double=10.0, ml.machine_memory=64424509440, ml.config_version=12.0.0, ml.max_jvm_size=32212254720, ml.allocated_processors=10, xpack.installed=true, transform.config_version=10.0.0, k8s_node_name=elk8-data-15}
Hot threads at 2024-03-10T16:33:50.981Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.3%, other=0.7%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-9][search_worker][T#11]'
10/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f2b3d4bc818.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f2b3d4bcf58.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.2%, other=0.8%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-9][search_worker][T#12]'
4/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:140)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f2b3d4bc818.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f2b3d4bcf58.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f2b3d4bc818.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f2b3d4bcf58.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f2b3d4bc818.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f2b3d4bcf58.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f2b3d4bc818.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f2b3d4bcf58.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.2%, other=0.8%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-9][search_worker][T#5]'
2/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:140)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f2b3d4bc818.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f2b3d4bcf58.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 26 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f2b3d4bc818.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f2b3d4bcf58.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f2b3d4bc818.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f2b3d4bcf58.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
4/10 snapshots sharing following 23 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f2b3d4bc818.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f2b3d4bcf58.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-cold-nodes-5}{-BNl7ehATsG9mcJCDW7ZzA}{BapTfUHTR7Cz_krePz7r9w}{eck-prod-es-data-cold-nodes-5}{10.233.97.40}{10.233.97.40:9300}{cf}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, k8s_node_name=elk8-data-02}
Hot threads at 2024-03-10T16:33:50.982Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.8%, other=0.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-5][search_worker][T#2]'
2/10 snapshots sharing following 27 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:154)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fc4254a1fe0.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fc4254a2720.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 27 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fc4254a1fe0.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fc4254a2720.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 29 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1248)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fc4254a1fe0.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fc4254a2720.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fc4254a1fe0.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fc4254a2720.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fc4254a1fe0.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fc4254a2720.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.6%, other=0.4%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-5][search_worker][T#11]'
2/10 snapshots sharing following 27 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fc4254a1fe0.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fc4254a2720.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
4/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:140)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fc4254a1fe0.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fc4254a2720.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 29 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fc4254a1fe0.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fc4254a2720.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fc4254a1fe0.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fc4254a2720.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.5%, other=0.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-cold-nodes-5][search_worker][T#13]'
6/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fc4254a1fe0.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fc4254a2720.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fc4254a1fe0.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fc4254a2720.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.packed.DirectMonotonicReader.get(DirectMonotonicReader.java:153)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1121)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fc4254a1fe0.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fc4254a2720.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-nodes-3}{hqRULs95TamvAKtgfTwaRg}{-69Z_WQ_S7qkP_SPT5ZnMA}{eck-prod-es-data-nodes-3}{10.233.92.171}{10.233.92.171:9300}{dhilrstw}{8.12.2}{7000099-8500010}{ml.allocated_processors=10, ml.max_jvm_size=32212254720, ml.config_version=12.0.0, ml.machine_memory=64424509440, ml.allocated_processors_double=10.0, transform.config_version=10.0.0, xpack.installed=true, k8s_node_name=elk8-data-17}
Hot threads at 2024-03-10T16:33:50.982Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.6% [cpu=100.6%, other=0.0%] (503.2ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-3][search_worker][T#5]'
4/10 snapshots sharing following 23 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f736d4ce300.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f736d4cea40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
5/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f736d4ce300.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f736d4cea40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f736d4ce300.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f736d4cea40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.6% [cpu=100.6%, other=0.0%] (502.8ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-3][search_worker][T#14]'
2/10 snapshots sharing following 27 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:154)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f736d4ce300.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f736d4cea40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f736d4ce300.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f736d4cea40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 23 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f736d4ce300.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f736d4cea40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 26 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f736d4ce300.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f736d4cea40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.7%, other=0.3%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-3][search_worker][T#2]'
7/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:140)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f736d4ce300.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f736d4cea40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 29 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1248)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f736d4ce300.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f736d4cea40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f736d4ce300.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f736d4cea40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-nodes-7}{yASHb2U0Sumohq0cEkxA_g}{GcSf3-v9RrGeNrt5_ViFLg}{eck-prod-es-data-nodes-7}{10.233.119.235}{10.233.119.235:9300}{dhilrstw}{8.12.2}{7000099-8500010}{ml.allocated_processors=10, ml.max_jvm_size=32212254720, ml.config_version=12.0.0, ml.allocated_processors_double=10.0, ml.machine_memory=64424509440, xpack.installed=true, transform.config_version=10.0.0, k8s_node_name=elk8-data-06}
Hot threads at 2024-03-10T16:33:50.982Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.5%, other=0.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-7][search_worker][T#7]'
4/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f6a954e8b18.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f6a954e9258.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
6/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f6a954e8b18.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f6a954e9258.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.1%, other=0.9%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-7][search_worker][T#2]'
3/10 snapshots sharing following 29 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefArray.get(BytesRefArray.java:107)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.get(BytesRefHash.java:129)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f6a954e8b18.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f6a954e9258.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
7/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f6a954e8b18.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f6a954e9258.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=98.8%, other=1.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-7][search_worker][T#13]'
9/10 snapshots sharing following 25 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f6a954e8b18.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f6a954e9258.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f6a954e8b18.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f6a954e9258.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-nodes-1}{H_EYW5lWQNayzc5YeJtyJA}{YF_wlE9CRbO5D2tRJQ-MNQ}{eck-prod-es-data-nodes-1}{10.233.126.171}{10.233.126.171:9300}{dhilrstw}{8.12.2}{7000099-8500010}{ml.allocated_processors=10, ml.max_jvm_size=32212254720, ml.config_version=12.0.0, ml.allocated_processors_double=10.0, ml.machine_memory=64424509440, xpack.installed=true, transform.config_version=10.0.0, k8s_node_name=elk8-data-01}
Hot threads at 2024-03-10T16:33:50.984Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.8%, other=0.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-1][search_worker][T#16]'
3/10 snapshots sharing following 23 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:388)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9cb944e308.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9cb944ea48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
7/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9cb944e308.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9cb944ea48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.8%, other=0.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-1][search_worker][T#6]'
5/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9cb944e308.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9cb944ea48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9cb944e308.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9cb944ea48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9cb944e308.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9cb944ea48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.packed.DirectMonotonicReader.get(DirectMonotonicReader.java:153)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1121)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9cb944e308.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9cb944ea48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=98.5%, other=1.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-1][search_worker][T#8]'
10/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9cb944e308.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9cb944ea48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-nodes-0}{CTP2lfUsQjuEKYepsv6qPQ}{9gZUhODIQ4ijBy7svJaybQ}{eck-prod-es-data-nodes-0}{10.233.110.172}{10.233.110.172:9300}{dhilrstw}{8.12.2}{7000099-8500010}{ml.machine_memory=64424509440, ml.max_jvm_size=32212254720, ml.config_version=12.0.0, ml.allocated_processors_double=10.0, ml.allocated_processors=10, transform.config_version=10.0.0, xpack.installed=true, k8s_node_name=elk8-data-14}
Hot threads at 2024-03-10T16:33:50.982Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.3%, other=0.7%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-0][search_worker][T#16]'
3/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:140)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f30ed4d5608.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f30ed4d5d48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
4/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f30ed4d5608.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f30ed4d5d48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 25 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f30ed4d5608.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f30ed4d5d48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f30ed4d5608.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f30ed4d5d48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.3%, other=0.7%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-0][search_worker][T#11]'
2/10 snapshots sharing following 25 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f30ed4d5608.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f30ed4d5d48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 23 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f30ed4d5608.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f30ed4d5d48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
5/10 snapshots sharing following 26 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f30ed4d5608.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f30ed4d5d48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.1%, other=0.9%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-0][search_worker][T#9]'
4/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:140)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f30ed4d5608.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f30ed4d5d48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.store.DataInput.readVInt(DataInput.java:121)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1248)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f30ed4d5608.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f30ed4d5d48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 23 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f30ed4d5608.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f30ed4d5d48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1253)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f30ed4d5608.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f30ed4d5d48.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-nodes-10}{FYSI3f1gSQa16i5nlOeysw}{1nG09g1DSRCQGAwsOWYQgQ}{eck-prod-es-data-nodes-10}{10.233.89.170}{10.233.89.170:9300}{dhilrstw}{8.12.2}{7000099-8500010}{ml.allocated_processors=10, ml.max_jvm_size=32212254720, ml.config_version=12.0.0, ml.machine_memory=64424509440, ml.allocated_processors_double=10.0, transform.config_version=10.0.0, xpack.installed=true, k8s_node_name=elk8-data-10}
Hot threads at 2024-03-10T16:33:50.982Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.7%, other=0.3%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-10][search_worker][T#4]'
2/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.store.DataInput.readVInt(DataInput.java:121)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1248)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5254b8e80.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5254b9e40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
4/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5254b8e80.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5254b9e40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
4/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5254b8e80.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5254b9e40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.7%, other=0.3%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-10][search_worker][T#1]'
2/10 snapshots sharing following 27 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:154)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5254b8e80.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5254b9e40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
5/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5254b8e80.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5254b9e40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5254b8e80.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5254b9e40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.4%, other=0.6%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-10][search_worker][T#12]'
4/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5254b8e80.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5254b9e40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
5/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5254b8e80.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5254b9e40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.packed.DirectMonotonicReader.get(DirectMonotonicReader.java:153)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1121)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007ff5254b8e80.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007ff5254b9e40.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-nodes-2}{gmY6bkGeRMaU0kQESLtY0A}{ZOteIxK7RyirGRApRwvL2w}{eck-prod-es-data-nodes-2}{10.233.117.234}{10.233.117.234:9300}{dhilrstw}{8.12.2}{7000099-8500010}{ml.allocated_processors_double=10.0, ml.machine_memory=64424509440, ml.config_version=12.0.0, ml.max_jvm_size=32212254720, ml.allocated_processors=10, xpack.installed=true, transform.config_version=10.0.0, k8s_node_name=elk8-data-09}
Hot threads at 2024-03-10T16:33:50.982Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=96.9%, other=3.1%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-2][search_worker][T#8]'
4/10 snapshots sharing following 28 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.get(BytesRefHash.java:129)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f358d4a8000.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f358d4a8960.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:140)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f358d4a8000.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f358d4a8960.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f358d4a8000.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f358d4a8960.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f358d4a8000.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f358d4a8960.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=96.9%, other=3.1%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-2][search_worker][T#15]'
3/10 snapshots sharing following 29 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f358d4a8000.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f358d4a8960.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 28 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.get(BytesRefHash.java:129)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f358d4a8000.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f358d4a8960.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 27 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:154)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f358d4a8000.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f358d4a8960.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f358d4a8000.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f358d4a8960.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f358d4a8000.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f358d4a8960.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=96.8%, other=3.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-2][search_worker][T#9]'
2/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.store.DataInput.readVInt(DataInput.java:121)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1248)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f358d4a8000.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f358d4a8960.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
5/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f358d4a8000.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f358d4a8960.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f358d4a8000.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f358d4a8960.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-nodes-5}{eE6E3iEnRpilJftmqiFaug}{-gLjT5qKQ8edjpz_vtidBw}{eck-prod-es-data-nodes-5}{10.233.71.215}{10.233.71.215:9300}{dhilrstw}{8.12.2}{7000099-8500010}{ml.machine_memory=64424509440, ml.allocated_processors_double=10.0, ml.allocated_processors=10, ml.max_jvm_size=32212254720, ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true, k8s_node_name=elk8-data-19}
Hot threads at 2024-03-10T16:33:50.982Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=93.0%, other=7.0%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-5][search_worker][T#5]'
2/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe0354d9f88.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe0354da6c8.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe0354d9f88.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe0354da6c8.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
5/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe0354d9f88.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe0354da6c8.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=93.0%, other=7.0%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-5][search_worker][T#12]'
7/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe0354d9f88.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe0354da6c8.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe0354d9f88.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe0354da6c8.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=92.9%, other=7.1%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-5][search_worker][T#8]'
5/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe0354d9f88.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe0354da6c8.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
5/10 snapshots sharing following 26 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007fe0354d9f88.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007fe0354da6c8.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-nodes-11}{6pvYmVZaSPySJ62Ob8AqnQ}{In9Z5SCUTCivhlvYNzYS1g}{eck-prod-es-data-nodes-11}{10.233.96.44}{10.233.96.44:9300}{dhilrstw}{8.12.2}{7000099-8500010}{ml.allocated_processors_double=10.0, ml.machine_memory=64424509440, ml.config_version=12.0.0, ml.max_jvm_size=32212254720, ml.allocated_processors=10, xpack.installed=true, transform.config_version=10.0.0, k8s_node_name=elk8-data-03}
Hot threads at 2024-03-10T16:33:50.983Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=99.6%, other=0.4%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-11][search_worker][T#14]'
2/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.store.DataInput.readVInt(DataInput.java:121)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1248)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9795487830.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9795489428.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 28 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.get(BytesRefHash.java:129)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9795487830.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9795489428.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9795487830.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9795489428.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9795487830.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9795489428.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.5%, other=0.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-11][search_worker][T#5]'
3/10 snapshots sharing following 28 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9795487830.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9795489428.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 28 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.get(BytesRefHash.java:129)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9795487830.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9795489428.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9795487830.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9795489428.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
unique snapshot
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9795487830.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9795489428.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=99.5%, other=0.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-11][search_worker][T#3]'
5/10 snapshots sharing following 28 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.get(BytesRefHash.java:129)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.set(BytesRefHash.java:161)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9795487830.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9795489428.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 27 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9795487830.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9795489428.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f9795487830.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f9795489428.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
::: {eck-prod-es-data-nodes-4}{YHFLPs-0STSZKNNKt9XhTA}{vdm3CI-UTjqx5Jkbpa9sRA}{eck-prod-es-data-nodes-4}{10.233.66.109}{10.233.66.109:9300}{dhilrstw}{8.12.2}{7000099-8500010}{ml.allocated_processors_double=10.0, ml.machine_memory=64424509440, ml.config_version=12.0.0, ml.max_jvm_size=32212254720, ml.allocated_processors=10, xpack.installed=true, transform.config_version=10.0.0, k8s_node_name=elk8-data-04}
Hot threads at 2024-03-10T16:33:50.982Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.0% [cpu=80.8%, other=19.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-4][search_worker][T#5]'
3/10 snapshots sharing following 26 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:201)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f7c494f1b10.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f7c494f2250.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
4/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:140)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1251)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f7c494f1b10.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f7c494f2250.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
3/10 snapshots sharing following 26 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f7c494f1b10.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f7c494f2250.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=79.1%, other=20.9%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-4][search_worker][T#11]'
3/10 snapshots sharing following 30 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.store.DataInput.readVInt(DataInput.java:121)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.decompressBlock(Lucene90DocValuesProducer.java:1248)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.next(Lucene90DocValuesProducer.java:1093)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$TermsDict.seekExact(Lucene90DocValuesProducer.java:1127)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f7c494f1b10.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f7c494f2250.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
5/10 snapshots sharing following 26 elements
app/org.apache.lucene.core@9.9.2/org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer$BaseSortedDocValues.lookupOrd(Lucene90DocValuesProducer.java:982)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.index.SingletonSortedSetDocValues.lookupOrd(SingletonSortedSetDocValues.java:95)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.index.fielddata.FieldData$9.nextValue(FieldData.java:313)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:389)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f7c494f1b10.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f7c494f2250.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
2/10 snapshots sharing following 25 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.BytesRefHash.add(BytesRefHash.java:206)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.BytesKeyedBucketOrds$FromSingle.add(BytesKeyedBucketOrds.java:110)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$StandardTermsResults.collectZeroDocEntriesIfNeeded(MapStringTermsAggregator.java:391)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f7c494f1b10.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f7c494f2250.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
100.0% [cpu=78.6%, other=21.5%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[eck-prod-es-data-nodes-4][search_worker][T#2]'
10/10 snapshots sharing following 22 elements
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator$ResultStrategy.buildAggregations(MapStringTermsAggregator.java:247)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.bucket.terms.MapStringTermsAggregator.buildAggregations(MapStringTermsAggregator.java:111)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.aggregation.ProfilingAggregator.buildAggregations(ProfilingAggregator.java:75)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.Aggregator.buildTopLevel(Aggregator.java:159)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.aggregations.AggregatorCollector.doPostCollection(AggregatorCollector.java:47)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.query.QueryPhaseCollector.doPostCollection(QueryPhaseCollector.java:379)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.profile.query.InternalProfileCollector.doPostCollection(InternalProfileCollector.java:83)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.doAggregationPostCollection(ContextIndexSearcher.java:386)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:375)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:345)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.search.internal.ContextIndexSearcher$$Lambda/0x00007f7c494f1b10.call(Unknown Source)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:118)
app/org.apache.lucene.core@9.9.2/org.apache.lucene.search.TaskExecutor$TaskGroup$$Lambda/0x00007f7c494f2250.call(Unknown Source)
java.base@21.0.2/java.util.concurrent.FutureTask.run(FutureTask.java:317)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
app/org.elasticsearch.server@8.12.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment