Skip to content

Instantly share code, notes, and snippets.

@ssouris

ssouris/HotThreadsResponse.log

Created May 18, 2020 20:23
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ssouris/9475140109fa12990d58c2ff97f40371 to your computer and use it in GitHub Desktop.
Save ssouris/9475140109fa12990d58c2ff97f40371 to your computer and use it in GitHub Desktop.
Download ZIP
Hot Threads API response
Raw
HotThreadsResponse.log
::: {node1.mycompany.net}{9GFXG3OWTsqz-dzQQblUmw}{UUtKg-5cSuaZuKtQAjfnMA}{10.48.112.217}{10.48.112.217:9300}{di}{aws_az=us-west-1b, xpack.installed=true}
Hot threads at 2020-05-18T20:16:02.896Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
6.1% (30.3ms out of 500ms) cpu usage by thread 'elasticsearch[node1.mycompany.net][write][T#2]'
unique snapshot
java.base@13.0.2/java.util.stream.Streams$ConcatSpliterator.<init>(Streams.java:705)
java.base@13.0.2/java.util.stream.Streams$ConcatSpliterator$OfRef.<init>(Streams.java:773)
java.base@13.0.2/java.util.stream.Stream.concat(Stream.java:1380)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ThreadContextStruct.putResponseHeaders(ThreadContext.java:522)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ThreadContextStruct.access$1900(ThreadContext.java:440)
app//org.elasticsearch.common.util.concurrent.ThreadContext.lambda$newStoredContext$3(ThreadContext.java:187)
app//org.elasticsearch.common.util.concurrent.ThreadContext$$Lambda$3753/0x00000008016f8840.close(Unknown Source)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:317)
app//org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)
app//org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
5.8% (29.1ms out of 500ms) cpu usage by thread 'elasticsearch[node1.mycompany.net][write][T#3]'
3/10 snapshots sharing following 5 elements
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
6/10 snapshots sharing following 10 elements
java.base@13.0.2/jdk.internal.misc.Unsafe.park(Native Method)
java.base@13.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:743)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:684)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1366)
app//org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
5.8% (28.8ms out of 500ms) cpu usage by thread 'elasticsearch[node1.mycompany.net][write][T#4]'
2/10 snapshots sharing following 5 elements
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
7/10 snapshots sharing following 10 elements
java.base@13.0.2/jdk.internal.misc.Unsafe.park(Native Method)
java.base@13.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:743)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:684)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1366)
app//org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
::: {node2.mycompany.net}{AsBUarcNQ1OvVqb6T1UycA}{25h3ZN1GQSuqAU7_jztZvw}{10.48.179.68}{10.48.179.68:9300}{di}{aws_az=us-west-1c, xpack.installed=true}
Hot threads at 2020-05-18T20:16:02.898Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.3% (501.3ms out of 500ms) cpu usage by thread 'elasticsearch[node2.mycompany.net][[ping-2020.05.18][1]: Lucene Merge Thread #574]'
7/10 snapshots sharing following 12 elements
app//org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
app//org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
app//org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
app//org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
app//org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:197)
app//org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:245)
app//org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:140)
app//org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4463)
app//org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4057)
app//org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
app//org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
app//org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
3/10 snapshots sharing following 9 elements
app//org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
app//org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:197)
app//org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:245)
app//org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:140)
app//org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4463)
app//org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4057)
app//org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
app//org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
app//org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
9.1% (45.6ms out of 500ms) cpu usage by thread 'elasticsearch[node2.mycompany.net][write][T#3]'
2/10 snapshots sharing following 177 elements
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:785)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:749)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:455)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:437)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:408)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:81)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:513)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:487)
app//org.elasticsearch.index.shard.IndexShard.lambda$innerAcquireReplicaOperationPermit$30(IndexShard.java:2984)
app//org.elasticsearch.index.shard.IndexShard$$Lambda$4638/0x000000080198e840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113)
app//org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285)
app//org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237)
app//org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$28(IndexShard.java:2923)
app//org.elasticsearch.index.shard.IndexShard$$Lambda$4636/0x0000000801989440.accept(Unknown Source)
app//org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:3023)
app//org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2922)
app//org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:849)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:585)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.action.support.replication.TransportReplicationAction.handleReplicaRequest(TransportReplicationAction.java:472)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$$Lambda$2902/0x0000000801532040.messageReceived(Unknown Source)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$$Lambda$3876/0x000000080177c040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4060/0x0000000801840c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3861/0x000000080177a040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4056/0x0000000801830c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3861/0x000000080177a040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4056/0x0000000801830c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3861/0x000000080177a040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4056/0x0000000801830c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3861/0x000000080177a040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4056/0x0000000801830c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3861/0x000000080177a040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71)
org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345)
org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4039/0x0000000801835440.accept(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616)
org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591)
app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:507)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:298)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4043/0x0000000801833840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$4045/0x0000000801833040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:554)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:251)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4047/0x0000000801832840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$4045/0x0000000801833040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:329)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:247)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4037/0x0000000801836040.getAsync(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:651)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:250)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4038/0x0000000801835840.getAsync(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:651)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:290)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4041/0x0000000801834040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:314)
org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:287)
org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:261)
org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4031/0x0000000801837840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4033/0x0000000801837040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:155)
org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:246)
org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123)
org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111)
org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile$$Lambda$3878/0x000000080177b840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$3880/0x000000080177b040.accept(Unknown Source)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$3881/0x000000080177b440.run(Unknown Source)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)
org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313)
app//org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)
app//org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
4/10 snapshots sharing following 10 elements
java.base@13.0.2/jdk.internal.misc.Unsafe.park(Native Method)
java.base@13.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:743)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:684)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1366)
app//org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
::: {node6.mycompany.net}{agmmOq3GQiCJrlbGCnDdEQ}{grd7bqu8TIC_zuzX5fHY3Q}{10.48.167.209}{10.48.167.209:9300}{di}{aws_az=us-west-1c, xpack.installed=true}
Hot threads at 2020-05-18T20:16:02.897Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
6.5% (32.3ms out of 500ms) cpu usage by thread 'elasticsearch[node6.mycompany.net][write][T#1]'
5/10 snapshots sharing following 2 elements
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
::: {master-node2.mycompany.net}{0e7F6OGiSey77QlnY94X8Q}{wOszHzfWRq-rgwN75Zfzew}{10.48.163.42}{10.48.163.42:9300}{m}{aws_az=us-west-1c, xpack.installed=true}
Hot threads at 2020-05-18T20:16:02.897Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {master-node1.mycompany.net}{6Z-n68w1QhevbZPGmHwxxA}{IMgwMCBDR0e60WsHzx853A}{10.48.101.175}{10.48.101.175:9300}{m}{aws_az=us-west-1b, xpack.installed=true}
Hot threads at 2020-05-18T20:16:02.897Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {node4.mycompany.net}{glKhJ5IrQCyZ7cjaRVghew}{DHl4_r1BQMibhP1Xk2tCiA}{10.48.168.59}{10.48.168.59:9300}{di}{aws_az=us-west-1c, xpack.installed=true}
Hot threads at 2020-05-18T20:16:02.897Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
5.1% (25.4ms out of 500ms) cpu usage by thread 'elasticsearch[node4.mycompany.net][write][T#4]'
2/10 snapshots sharing following 5 elements
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
6/10 snapshots sharing following 10 elements
java.base@13.0.2/jdk.internal.misc.Unsafe.park(Native Method)
java.base@13.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:743)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:684)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1366)
app//org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
::: {master-node1.mycompany.net}{hjLZNZIORUO_oPf3AkSmUw}{DNEW6nkOT46GUhAE2VKP1Q}{10.64.44.98}{10.64.44.98:9300}{m}{aws_az=us-west-2a, xpack.installed=true}
Hot threads at 2020-05-18T20:16:02.907Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {node3.mycompany.net}{IihvXw6FS5WFQO6X3m7gYA}{qfGjYas8TUykoFH-sMpoXA}{10.48.120.183}{10.48.120.183:9300}{di}{aws_az=us-west-1b, xpack.installed=true}
Hot threads at 2020-05-18T20:16:02.896Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
7.2% (35.9ms out of 500ms) cpu usage by thread 'elasticsearch[node3.mycompany.net][write][T#4]'
2/10 snapshots sharing following 170 elements
app//org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:487)
app//org.elasticsearch.index.shard.IndexShard.lambda$innerAcquireReplicaOperationPermit$30(IndexShard.java:2984)
app//org.elasticsearch.index.shard.IndexShard$$Lambda$4588/0x0000000801969840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113)
app//org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285)
app//org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237)
app//org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$28(IndexShard.java:2923)
app//org.elasticsearch.index.shard.IndexShard$$Lambda$4587/0x0000000801969040.accept(Unknown Source)
app//org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:3023)
app//org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2922)
app//org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:849)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:585)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.action.support.replication.TransportReplicationAction.handleReplicaRequest(TransportReplicationAction.java:472)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$$Lambda$2901/0x0000000801575440.messageReceived(Unknown Source)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$$Lambda$3876/0x0000000801784040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4075/0x0000000801843840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3862/0x0000000801781440.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4072/0x0000000801842c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3862/0x0000000801781440.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4072/0x0000000801842c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3862/0x0000000801781440.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4072/0x0000000801842c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3862/0x0000000801781440.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4072/0x0000000801842c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3862/0x0000000801781440.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)
org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345)
org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4059/0x0000000801832c40.accept(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616)
org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591)
app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:507)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:298)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4063/0x0000000801840840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$4065/0x0000000801841040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:554)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:251)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4067/0x0000000801841840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$4065/0x0000000801841040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:329)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:247)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4057/0x0000000801833840.getAsync(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:651)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:250)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4058/0x0000000801833040.getAsync(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:651)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:290)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4061/0x0000000801840040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:314)
org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:287)
org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:261)
org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4053/0x0000000801834840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4055/0x0000000801834040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:155)
org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:246)
org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123)
org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111)
org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile$$Lambda$3878/0x0000000801783840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$3880/0x0000000801783040.accept(Unknown Source)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$3881/0x0000000801783440.run(Unknown Source)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)
org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313)
app//org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)
app//org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
7/10 snapshots sharing following 10 elements
java.base@13.0.2/jdk.internal.misc.Unsafe.park(Native Method)
java.base@13.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:743)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:684)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1366)
app//org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
6.8% (34.1ms out of 500ms) cpu usage by thread 'elasticsearch[node3.mycompany.net][write][T#2]'
4/10 snapshots sharing following 2 elements
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
::: {node5.mycompany.net}{Vdh1RxdrT0aZZeME_Dk2dQ}{VNTSBr1fTPazSejYJqua0g}{10.48.105.165}{10.48.105.165:9300}{di}{aws_az=us-west-1b, xpack.installed=true}
Hot threads at 2020-05-18T20:16:02.896Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
21.3% (106.5ms out of 500ms) cpu usage by thread 'elasticsearch[node5.mycompany.net][refresh][T#1]'
7/10 snapshots sharing following 25 elements
app//org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:494)
app//org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:297)
app//org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:272)
app//org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
app//org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
app//org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:165)
app//org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:66)
app//org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:40)
app//org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
app//org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
app//org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:344)
app//org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:326)
app//org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
app//org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
app//org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1611)
app//org.elasticsearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1590)
app//org.elasticsearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:3236)
app//org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:827)
app//org.elasticsearch.index.IndexService.access$200(IndexService.java:108)
app//org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:959)
app//org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:144)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
2/10 snapshots sharing following 26 elements
app//org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
app//org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526)
app//org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:297)
app//org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:272)
app//org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
app//org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
app//org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:165)
app//org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:66)
app//org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:40)
app//org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
app//org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
app//org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:344)
app//org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:326)
app//org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
app//org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
app//org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1611)
app//org.elasticsearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1590)
app//org.elasticsearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:3236)
app//org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:827)
app//org.elasticsearch.index.IndexService.access$200(IndexService.java:108)
app//org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:959)
app//org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:144)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
unique snapshot
java.base@13.0.2/sun.nio.fs.UnixNativeDispatcher.unlink0(Native Method)
java.base@13.0.2/sun.nio.fs.UnixNativeDispatcher.unlink(UnixNativeDispatcher.java:156)
java.base@13.0.2/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:236)
java.base@13.0.2/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
java.base@13.0.2/java.nio.file.Files.delete(Files.java:1145)
app//org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:370)
app//org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:339)
app//org.apache.lucene.store.FilterDirectory.deleteFile(FilterDirectory.java:63)
app//org.elasticsearch.index.store.ByteSizeCachingDirectory.deleteFile(ByteSizeCachingDirectory.java:176)
app//org.apache.lucene.store.FilterDirectory.deleteFile(FilterDirectory.java:63)
app//org.elasticsearch.index.store.Store$StoreDirectory.deleteFile(Store.java:713)
app//org.elasticsearch.index.store.Store$StoreDirectory.deleteFile(Store.java:718)
app//org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
app//org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
app//org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
app//org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:589)
app//org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:620)
app//org.apache.lucene.index.IndexWriter.decRefDeleter(IndexWriter.java:5106)
app//org.apache.lucene.index.StandardDirectoryReader.lambda$doClose$1(StandardDirectoryReader.java:377)
app//org.apache.lucene.index.StandardDirectoryReader$$Lambda$306/0x0000000800cd6840.close(Unknown Source)
app//org.apache.lucene.index.StandardDirectoryReader.doClose(StandardDirectoryReader.java:391)
app//org.apache.lucene.index.IndexReader.decRef(IndexReader.java:244)
app//org.apache.lucene.index.IndexReader.close(IndexReader.java:385)
app//org.apache.lucene.index.FilterDirectoryReader.doClose(FilterDirectoryReader.java:142)
app//org.apache.lucene.index.IndexReader.decRef(IndexReader.java:244)
app//org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.decRef(InternalEngine.java:379)
app//org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.decRef(InternalEngine.java:326)
app//org.apache.lucene.search.ReferenceManager.release(ReferenceManager.java:274)
app//org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:189)
app//org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
app//org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1611)
app//org.elasticsearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1590)
app//org.elasticsearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:3236)
app//org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:827)
app//org.elasticsearch.index.IndexService.access$200(IndexService.java:108)
app//org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:959)
app//org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:144)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
18.1% (90.5ms out of 500ms) cpu usage by thread 'elasticsearch[node5.mycompany.net][write][T#2]'
6/10 snapshots sharing following 190 elements
app//org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:468)
app//org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:555)
app//org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:402)
app//org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:480)
app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594)
app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213)
app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1157)
app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1094)
app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:934)
app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:813)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:785)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:742)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:267)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:157)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:202)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:114)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:81)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:895)
app//org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:109)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:374)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:297)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction$$Lambda$4362/0x00000008018e7040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$24(IndexShard.java:2791)
app//org.elasticsearch.index.shard.IndexShard$$Lambda$4364/0x00000008018f0040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113)
app//org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285)
app//org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237)
app//org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2765)
app//org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:836)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:293)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:256)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$$Lambda$2899/0x000000080156cc40.messageReceived(Unknown Source)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$$Lambda$3859/0x0000000801786040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4064/0x0000000801843040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3844/0x000000080176b040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4062/0x0000000801842840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3844/0x000000080176b040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4062/0x0000000801842840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3844/0x000000080176b040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4062/0x0000000801842840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3844/0x000000080176b040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4062/0x0000000801842840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3844/0x000000080176b040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71)
org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345)
org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4049/0x000000080183f040.accept(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616)
org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591)
app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:507)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:298)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4053/0x0000000801840440.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$4055/0x0000000801840c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:554)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:251)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4057/0x0000000801841440.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$4055/0x0000000801840c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:329)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:247)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4047/0x000000080183dc40.getAsync(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:651)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:250)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4048/0x000000080183e440.getAsync(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:651)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:290)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4051/0x000000080183fc40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:314)
org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:287)
org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:261)
org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4043/0x000000080183cc40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4045/0x000000080183d440.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:155)
org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:246)
org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123)
org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111)
org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile$$Lambda$3861/0x0000000801785840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$3863/0x0000000801785040.accept(Unknown Source)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$3864/0x0000000801785440.run(Unknown Source)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)
org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313)
app//org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)
app//org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:750)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
3/10 snapshots sharing following 185 elements
app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213)
app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1157)
app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1094)
app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:934)
app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:813)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:785)
app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:742)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:267)
app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:157)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:202)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:114)
app//org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:81)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:895)
app//org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:109)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:374)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:297)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction$$Lambda$4362/0x00000008018e7040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$24(IndexShard.java:2791)
app//org.elasticsearch.index.shard.IndexShard$$Lambda$4364/0x00000008018f0040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113)
app//org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285)
app//org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237)
app//org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2765)
app//org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:836)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:293)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:256)
app//org.elasticsearch.action.support.replication.TransportReplicationAction$$Lambda$2899/0x000000080156cc40.messageReceived(Unknown Source)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$$Lambda$3859/0x0000000801786040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:344)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4064/0x0000000801843040.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3844/0x000000080176b040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
org.elasticsearch.xpack.security.authz.interceptor.SearchRequestInterceptor.intercept(SearchRequestInterceptor.java:19)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4062/0x0000000801842840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3844/0x000000080176b040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:61)
org.elasticsearch.xpack.security.authz.interceptor.UpdateRequestInterceptor.intercept(UpdateRequestInterceptor.java:23)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4062/0x0000000801842840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3844/0x000000080176b040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.IndicesAliasesRequestInterceptor.intercept(IndicesAliasesRequestInterceptor.java:102)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4062/0x0000000801842840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3844/0x000000080176b040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:82)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:339)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4062/0x0000000801842840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
app//org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.ListenableFuture$$Lambda$3844/0x000000080176b040.accept(Unknown Source)
java.base@13.0.2/java.util.ArrayList.forEach(ArrayList.java:1507)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98)
app//org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144)
app//org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127)
app//org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62)
app//org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40)
org.elasticsearch.xpack.security.authz.interceptor.BulkShardRequestInterceptor.intercept(BulkShardRequestInterceptor.java:71)
org.elasticsearch.xpack.security.authz.AuthorizationService.runRequestInterceptors(AuthorizationService.java:345)
org.elasticsearch.xpack.security.authz.AuthorizationService.handleIndexActionAuthorizationResult(AuthorizationService.java:322)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$9(AuthorizationService.java:263)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4049/0x000000080183f040.accept(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:616)
org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:591)
app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
org.elasticsearch.xpack.security.authz.RBACEngine.buildIndicesAccessControl(RBACEngine.java:507)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$3(RBACEngine.java:298)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4053/0x0000000801840440.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$4055/0x0000000801840c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService.resolveIndexNames(AuthorizationService.java:554)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$6(AuthorizationService.java:251)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4057/0x0000000801841440.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.lambda$getAsync$0(AuthorizationService.java:653)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier$$Lambda$4055/0x0000000801840c40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.RBACEngine.loadAuthorizedIndices(RBACEngine.java:329)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$5(AuthorizationService.java:247)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4047/0x000000080183dc40.getAsync(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:651)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$8(AuthorizationService.java:250)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4048/0x000000080183e440.getAsync(Unknown Source)
org.elasticsearch.xpack.security.authz.AuthorizationService$CachingAsyncSupplier.getAsync(AuthorizationService.java:651)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:290)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4051/0x000000080183fc40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:314)
org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:287)
org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:261)
org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:227)
org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:193)
org.elasticsearch.xpack.security.authz.AuthorizationService$$Lambda$4043/0x000000080183cc40.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
app//org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:117)
org.elasticsearch.xpack.security.authz.RBACEngine$$Lambda$4045/0x000000080183d440.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.roles(CompositeRolesStore.java:155)
org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:246)
org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:123)
org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:111)
org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:195)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile$$Lambda$3861/0x0000000801785840.accept(Unknown Source)
app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$3863/0x0000000801785040.accept(Unknown Source)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator$$Lambda$3864/0x0000000801785440.run(Unknown Source)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245)
org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196)
org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139)
org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121)
org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313)
app//org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)
app//org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:750)
app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
unique snapshot
java.base@13.0.2/jdk.internal.misc.Unsafe.park(Native Method)
java.base@13.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:743)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:684)
java.base@13.0.2/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1366)
app//org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
7.0% (35.1ms out of 500ms) cpu usage by thread 'elasticsearch[node5.mycompany.net][write][T#3]'
8/10 snapshots sharing following 2 elements
java.base@13.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@13.0.2/java.lang.Thread.run(Thread.java:830)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment