Skip to content

Instantly share code, notes, and snippets.

@nikhil-bhat

nikhil-bhat/hot thread api Secret

Created September 25, 2019 09:22
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nikhil-bhat/8e519502599d3ebced308e5f744294de to your computer and use it in GitHub Desktop.
Save nikhil-bhat/8e519502599d3ebced308e5f744294de to your computer and use it in GitHub Desktop.
Download ZIP
hot thread api output -2
Raw
hot thread api
::: {redacted-master-03}{65xtKPd6RdeVTH9IXcpGTA}{AzCq9gK3QRyUe5TcpDxHWw}{x.y.63.212}{x.y.63.212:9300}{ml.machine_memory=7673597952, rack=us-central1-c, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:53.719Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {redacted-master-02}{GJZ25KnDTF-cX_Uxow5dqQ}{iK7UeMlnSbKRUk_8Z11nhQ}{x.y.63.211}{x.y.63.211:9300}{ml.machine_memory=7673597952, rack=us-central1-b, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:53.717Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {redacted-data-04}{JDNpGwn9TWa8c7rudyVKdA}{XCuhmnZGQguw4XWKqxgU_w}{x.y.63.216}{x.y.63.216:9300}{ml.machine_memory=67388059648, rack=us-central1-b, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:53.716Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
98.4% (492ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-04][[redacted-2019.09.19][0]: Lucene Merge Thread #77300]'
6/10 snapshots sharing following 15 elements
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73)
org.apache.lucene.codecs.compressing.LZ4.encodeLiterals(LZ4.java:155)
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:168)
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
2/10 snapshots sharing following 12 elements
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
2/10 snapshots sharing following 8 elements
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:641)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
::: {redacted-data-05}{YICkD-88RMORobFtFIaKew}{xkMgubP3TW6lbg90VzsatQ}{x.y.48.124}{x.y.48.124:9300}{ml.machine_memory=67388059648, rack=us-central1-c, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:53.716Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
84.8% (423.8ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-05][http_server_worker][T#10]'
4/10 snapshots sharing following 96 elements
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntry.getValue(Collections.java:1747)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.lambda$getResolvedIndexPattern$0(ConfigModel.java:646)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern$$Lambda$3201/669949651.test(Unknown Source)
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.lambda$entryConsumer$0(Collections.java:1575)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$$Lambda$3180/1895063989.accept(Unknown Source)
java.util.TreeMap$EntrySpliterator.forEachRemaining(TreeMap.java:2969)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntrySetSpliterator.forEachRemaining(Collections.java:1600)
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:647)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$4251/358510498.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
6/10 snapshots sharing following 82 elements
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$4251/358510498.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
12.5% (62.3ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-05][http_server_worker][T#1]'
5/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
::: {redacted-data-02}{BeV7xpVZRrS2zuRC4kFXjg}{sYBSb_Y1QOy0QQ1ET5VGng}{x.y.63.214}{x.y.63.214:9300}{ml.machine_memory=67388059648, rack=us-central1-a, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:53.716Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
85.6% (428.1ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-02][http_server_worker][T#14]'
3/10 snapshots sharing following 89 elements
java.util.regex.Pattern.<init>(Pattern.java:1351)
java.util.regex.Pattern.compile(Pattern.java:1054)
java.lang.String.replace(String.java:2239)
com.floragunn.searchguard.sgconf.ConfigModel.replaceProperties(ConfigModel.java:815)
com.floragunn.searchguard.sgconf.ConfigModel.access$1700(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getUnresolvedIndexPattern(ConfigModel.java:638)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:642)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$4223/1173467396.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
3/10 snapshots sharing following 96 elements
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntry.getValue(Collections.java:1747)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.lambda$getResolvedIndexPattern$0(ConfigModel.java:646)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern$$Lambda$3173/740219504.test(Unknown Source)
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.lambda$entryConsumer$0(Collections.java:1575)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$$Lambda$3149/2283194.accept(Unknown Source)
java.util.TreeMap$EntrySpliterator.forEachRemaining(TreeMap.java:2969)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntrySetSpliterator.forEachRemaining(Collections.java:1600)
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:647)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$4223/1173467396.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 85 elements
com.floragunn.searchguard.sgconf.ConfigModel.access$1700(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getUnresolvedIndexPattern(ConfigModel.java:638)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:642)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$4223/1173467396.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 88 elements
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:647)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$4223/1173467396.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
::: {redacted-data-08}{9tnGh4atRmGsXmNyyr5a0w}{r7M2rV8tR3yc7XKYTKAOPw}{x.y.48.127}{x.y.48.127:9300}{ml.machine_memory=67388059648, rack=us-central1-f, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true, data=hot}
Hot threads at 2019-09-25T08:16:53.768Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
28.6% (142.9ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-08][refresh][T#1]'
2/10 snapshots sharing following 28 elements
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:719)
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:493)
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259)
org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:326)
org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:301)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1639)
org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1620)
org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:967)
org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:738)
org.elasticsearch.index.IndexService.access$500(IndexService.java:98)
org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:867)
org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:141)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
16.9% (84.3ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-08][write][T#4]'
8/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
16.3% (81.5ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-08][refresh][T#4]'
2/10 snapshots sharing following 37 elements
org.apache.lucene.util.packed.DirectMonotonicWriter.<init>(DirectMonotonicWriter.java:56)
org.apache.lucene.util.packed.DirectMonotonicWriter.getInstance(DirectMonotonicWriter.java:136)
org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addTermsDict(Lucene70DocValuesConsumer.java:448)
org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.doAddSortedField(Lucene70DocValuesConsumer.java:437)
org.apache.lucene.codecs.lucene70.Lucene70DocValuesConsumer.addSortedSetField(Lucene70DocValuesConsumer.java:577)
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedSetField(PerFieldDocValuesFormat.java:129)
org.apache.lucene.index.SortedSetDocValuesWriter.flush(SortedSetDocValuesWriter.java:221)
org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:251)
org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:135)
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:719)
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:493)
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259)
org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:326)
org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:301)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1639)
org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1620)
org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:967)
org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:738)
org.elasticsearch.index.IndexService.access$500(IndexService.java:98)
org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:867)
org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:141)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
4/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {redacted-master-01}{jWZbC6rSSa-NMnPPWg6h-A}{LSa-ClFSR3ybfpKVlj0XCA}{x.y.63.210}{x.y.63.210:9300}{ml.machine_memory=7673556992, rack=us-central1-a, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:53.728Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {redacted-data-06}{giZ9Kn8tTY-b8T5v_WQ66g}{TS27dQZARI6Vblt1pC3vHQ}{x.y.48.125}{x.y.48.125:9300}{ml.machine_memory=67388059648, rack=us-central1-c, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:53.718Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
15.3% (76.3ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-06][transport_worker][T#14]'
3/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
9.1% (45.5ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-06][write][T#3]'
7/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {redacted-data-07}{2TSbcsZwTeyq2by30_GKjw}{eCvi7XZKTKKgfkj8Kni_vA}{x.y.48.126}{x.y.48.126:9300}{ml.machine_memory=67388059648, rack=us-central1-f, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:53.717Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.6% (502.9ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-07][[redacted-2019.09.19][0]: Lucene Merge Thread #35524]'
10/10 snapshots sharing following 7 elements
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
83.3% (416.3ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-07][write][T#1]'
7/10 snapshots sharing following 42 elements
java.lang.Object.notifyAll(Native Method)
org.elasticsearch.index.seqno.LocalCheckpointTracker.updateCheckpoint(LocalCheckpointTracker.java:209)
org.elasticsearch.index.seqno.LocalCheckpointTracker.markSeqNoAsCompleted(LocalCheckpointTracker.java:111)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:949)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:824)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:744)
org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:454)
org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$3815/1861827056.get(Unknown Source)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:477)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:452)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1050)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1028)
org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:424)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:370)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction$$Lambda$3743/852085019.accept(Unknown Source)
org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:987)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:369)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:324)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:311)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:169)
com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:139)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:91)
com.floragunn.searchguard.SearchGuardPlugin$7$1.messageReceived(SearchGuardPlugin.java:634)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 38 elements
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:824)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:744)
org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:454)
org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$3815/1861827056.get(Unknown Source)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:477)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:452)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1050)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1028)
org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:424)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:370)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction$$Lambda$3743/852085019.accept(Unknown Source)
org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:987)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:369)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:324)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:311)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:169)
com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:139)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:91)
com.floragunn.searchguard.SearchGuardPlugin$7$1.messageReceived(SearchGuardPlugin.java:634)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
unique snapshot
org.elasticsearch.common.xcontent.json.JsonXContentParser.currentToken(JsonXContentParser.java:62)
org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:227)
org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$7.parse(NumberFieldMapper.java:714)
org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$7.parse(NumberFieldMapper.java:685)
org.elasticsearch.index.mapper.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:1050)
org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:297)
org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:488)
org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:616)
org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:410)
org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384)
org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485)
org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505)
org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:395)
org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384)
org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485)
org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505)
org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:395)
org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384)
org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485)
org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505)
org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:395)
org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384)
org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:96)
org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69)
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:281)
org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:799)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:775)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:744)
org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:454)
org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$3815/1861827056.get(Unknown Source)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:477)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:452)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1050)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1028)
org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:424)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:370)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction$$Lambda$3743/852085019.accept(Unknown Source)
org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:987)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:369)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:324)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:311)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:169)
com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:139)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:91)
com.floragunn.searchguard.SearchGuardPlugin$7$1.messageReceived(SearchGuardPlugin.java:634)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {redacted-data-01}{mbqlTDPTToekF5-xTz7Z9g}{M8j__xJiQC6o1tohgCRKfw}{x.y.63.213}{x.y.63.213:9300}{ml.machine_memory=67388059648, rack=us-central1-a, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
Hot threads at 2019-09-25T08:16:53.716Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
86.3% (431.3ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-01][[redacted-2019.09.19][1]: Lucene Merge Thread #32794]'
2/10 snapshots sharing following 14 elements
org.apache.lucene.codecs.compressing.LZ4.encodeLiterals(LZ4.java:155)
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:168)
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
4/10 snapshots sharing following 14 elements
org.apache.lucene.codecs.compressing.LZ4.encodeLiterals(LZ4.java:147)
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:168)
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
2/10 snapshots sharing following 9 elements
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
2/10 snapshots sharing following 13 elements
org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:133)
org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.doReset(CompressingStoredFieldsReader.java:483)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.reset(CompressingStoredFieldsReader.java:389)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.document(CompressingStoredFieldsReader.java:568)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:641)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
85.9% (429.6ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-01][http_server_worker][T#13]'
2/10 snapshots sharing following 96 elements
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntry.getValue(Collections.java:1747)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.lambda$getResolvedIndexPattern$0(ConfigModel.java:646)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern$$Lambda$3051/1112143682.test(Unknown Source)
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.lambda$entryConsumer$0(Collections.java:1575)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$$Lambda$3042/1872138376.accept(Unknown Source)
java.util.TreeMap$EntrySpliterator.forEachRemaining(TreeMap.java:2969)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntrySetSpliterator.forEachRemaining(Collections.java:1600)
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:647)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$3992/545286910.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 86 elements
org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:168)
org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:147)
org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:107)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:650)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$3992/545286910.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
4/10 snapshots sharing following 83 elements
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:647)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$3992/545286910.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 89 elements
java.util.regex.Pattern.<init>(Pattern.java:1351)
java.util.regex.Pattern.compile(Pattern.java:1054)
java.lang.String.replace(String.java:2239)
com.floragunn.searchguard.sgconf.ConfigModel.replaceProperties(ConfigModel.java:814)
com.floragunn.searchguard.sgconf.ConfigModel.access$1700(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getUnresolvedIndexPattern(ConfigModel.java:638)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:642)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$3992/545286910.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
::: {redacted-data-03}{2KdrOqE7QzSfjzXxFunNmw}{TaW7U8JDQE66oDyXSRVecw}{x.y.63.215}{x.y.63.215:9300}{ml.machine_memory=67388059648, rack=us-central1-b, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:53.717Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
88.7% (443.3ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-03][[redacted-2019.09.19][1]: Lucene Merge Thread #69876]'
9/10 snapshots sharing following 16 elements
org.elasticsearch.common.lucene.store.FilterIndexOutput.writeBytes(FilterIndexOutput.java:59)
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73)
org.apache.lucene.codecs.compressing.LZ4.encodeLiterals(LZ4.java:155)
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:168)
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
unique snapshot
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
38.6% (192.9ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-03][write][T#6]'
2/10 snapshots sharing following 51 elements
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:207)
java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
java.nio.channels.Channels.writeFully(Channels.java:101)
java.nio.channels.Channels.access$000(Channels.java:61)
java.nio.channels.Channels$1.write(Channels.java:174)
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
org.elasticsearch.common.bytes.BytesReference.writeTo(BytesReference.java:90)
org.elasticsearch.index.translog.TranslogWriter.add(TranslogWriter.java:163)
org.elasticsearch.index.translog.Translog.add(Translog.java:545)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:932)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:824)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:744)
org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:454)
org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$3923/805585702.get(Unknown Source)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:477)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:452)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1050)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1028)
org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:424)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:370)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction$$Lambda$3855/2014971952.accept(Unknown Source)
org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:987)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:369)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:324)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:311)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:169)
com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:139)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:91)
com.floragunn.searchguard.SearchGuardPlugin$7$1.messageReceived(SearchGuardPlugin.java:634)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
8/10 snapshots sharing following 38 elements
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:824)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:744)
org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:454)
org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$3923/805585702.get(Unknown Source)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:477)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:452)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1050)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1028)
org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:424)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:370)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction$$Lambda$3855/2014971952.accept(Unknown Source)
org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:987)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:369)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:324)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:311)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:169)
com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:139)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:91)
com.floragunn.searchguard.SearchGuardPlugin$7$1.messageReceived(SearchGuardPlugin.java:634)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment