Skip to content

Instantly share code, notes, and snippets.

@nikhil-bhat

nikhil-bhat/hot threadapi start Secret

Last active September 25, 2019 09:21
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nikhil-bhat/b86983baab49cf6fd19732c82a951b33 to your computer and use it in GitHub Desktop.
Save nikhil-bhat/b86983baab49cf6fd19732c82a951b33 to your computer and use it in GitHub Desktop.
Download ZIP
Hot thread api output when index pattern refresh is initiated through kibana
Raw
hot threadapi start
::: {redacted-master-03}{65xtKPd6RdeVTH9IXcpGTA}{AzCq9gK3QRyUe5TcpDxHWw}{x.y.63.212}{x.y.63.212:9300}{ml.machine_memory=7673597952, rack=us-central1-c, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:15.903Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {redacted-master-02}{GJZ25KnDTF-cX_Uxow5dqQ}{iK7UeMlnSbKRUk_8Z11nhQ}{x.y.63.211}{x.y.63.211:9300}{ml.machine_memory=7673597952, rack=us-central1-b, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:15.902Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {redacted-data-04}{JDNpGwn9TWa8c7rudyVKdA}{XCuhmnZGQguw4XWKqxgU_w}{x.y.63.216}{x.y.63.216:9300}{ml.machine_memory=67388059648, rack=us-central1-b, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:15.901Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
98.3% (491.2ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-04][[redacted-2019.09.19][0]: Lucene Merge Thread #77300]'
10/10 snapshots sharing following 12 elements
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
::: {redacted-data-05}{YICkD-88RMORobFtFIaKew}{xkMgubP3TW6lbg90VzsatQ}{x.y.48.124}{x.y.48.124:9300}{ml.machine_memory=67388059648, rack=us-central1-c, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:15.901Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
80.9% (404.6ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-05][http_server_worker][T#10]'
3/10 snapshots sharing following 96 elements
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntry.getValue(Collections.java:1747)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.lambda$getResolvedIndexPattern$0(ConfigModel.java:646)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern$$Lambda$3201/669949651.test(Unknown Source)
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.lambda$entryConsumer$0(Collections.java:1575)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$$Lambda$3180/1895063989.accept(Unknown Source)
java.util.TreeMap$EntrySpliterator.forEachRemaining(TreeMap.java:2969)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntrySetSpliterator.forEachRemaining(Collections.java:1600)
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:647)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$4251/358510498.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 90 elements
java.util.regex.Pattern.compile(Pattern.java:1694)
java.util.regex.Pattern.<init>(Pattern.java:1351)
java.util.regex.Pattern.compile(Pattern.java:1054)
java.lang.String.replace(String.java:2239)
com.floragunn.searchguard.sgconf.ConfigModel.replaceProperties(ConfigModel.java:814)
com.floragunn.searchguard.sgconf.ConfigModel.access$1700(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getUnresolvedIndexPattern(ConfigModel.java:638)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:642)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$4251/358510498.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
5/10 snapshots sharing following 82 elements
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$4251/358510498.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
9.2% (46.1ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-05][write][T#8]'
4/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {redacted-data-02}{BeV7xpVZRrS2zuRC4kFXjg}{sYBSb_Y1QOy0QQ1ET5VGng}{x.y.63.214}{x.y.63.214:9300}{ml.machine_memory=67388059648, rack=us-central1-a, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:15.902Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
14.6% (72.8ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-02][write][T#3]'
2/10 snapshots sharing following 5 elements
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
6/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
13.0% (64.9ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-02][write][T#1]'
4/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {redacted-data-08}{9tnGh4atRmGsXmNyyr5a0w}{r7M2rV8tR3yc7XKYTKAOPw}{x.y.48.127}{x.y.48.127:9300}{ml.machine_memory=67388059648, rack=us-central1-f, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true, data=hot}
Hot threads at 2019-09-25T08:16:15.902Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
84.0% (419.8ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-08][[redacted-2019.09.24][0]: Lucene Merge Thread #42328]'
10/10 snapshots sharing following 8 elements
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:152)
org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:182)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:126)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
24.3% (121.6ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-08][write][T#1]'
9/10 snapshots sharing following 37 elements
java.lang.Object.notifyAll(Native Method)
org.elasticsearch.index.seqno.LocalCheckpointTracker.updateCheckpoint(LocalCheckpointTracker.java:209)
org.elasticsearch.index.seqno.LocalCheckpointTracker.markSeqNoAsCompleted(LocalCheckpointTracker.java:111)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:949)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:824)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:751)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:416)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:384)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:371)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:79)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:636)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:596)
org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2761)
org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2739)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$17(IndexShard.java:2699)
org.elasticsearch.index.shard.IndexShard$$Lambda$3347/1179103098.accept(Unknown Source)
org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:2806)
org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2698)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:1000)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:706)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:579)
org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:558)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:169)
com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:260)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:141)
com.floragunn.searchguard.SearchGuardPlugin$7$1.messageReceived(SearchGuardPlugin.java:634)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
unique snapshot
java.util.zip.Deflater.deflateBytes(Native Method)
java.util.zip.Deflater.deflate(Deflater.java:444)
java.util.zip.Deflater.deflate(Deflater.java:366)
org.apache.lucene.codecs.compressing.CompressionMode$DeflateCompressor.compress(CompressionMode.java:275)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.index.StoredFieldsConsumer.finishDocument(StoredFieldsConsumer.java:59)
org.apache.lucene.index.DefaultIndexingChain.finishStoredFields(DefaultIndexingChain.java:366)
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:402)
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1608)
org.elasticsearch.index.engine.InternalEngine.updateDocs(InternalEngine.java:1282)
org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1105)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:924)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:824)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:751)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:416)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:384)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:371)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:79)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:636)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:596)
org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2761)
org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2739)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$17(IndexShard.java:2699)
org.elasticsearch.index.shard.IndexShard$$Lambda$3347/1179103098.accept(Unknown Source)
org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:2806)
org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2698)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:1000)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:706)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:579)
org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:558)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:169)
com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:260)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:141)
com.floragunn.searchguard.SearchGuardPlugin$7$1.messageReceived(SearchGuardPlugin.java:634)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {redacted-master-01}{jWZbC6rSSa-NMnPPWg6h-A}{LSa-ClFSR3ybfpKVlj0XCA}{x.y.63.210}{x.y.63.210:9300}{ml.machine_memory=7673556992, rack=us-central1-a, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:15.903Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {redacted-data-06}{giZ9Kn8tTY-b8T5v_WQ66g}{TS27dQZARI6Vblt1pC3vHQ}{x.y.48.125}{x.y.48.125:9300}{ml.machine_memory=67388059648, rack=us-central1-c, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:15.903Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
12.3% (61.2ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-06][refresh][T#4]'
2/10 snapshots sharing following 28 elements
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:469)
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:554)
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:719)
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:493)
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:294)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:259)
org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:326)
org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:301)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1639)
org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1620)
org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:967)
org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:738)
org.elasticsearch.index.IndexService.access$500(IndexService.java:98)
org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:867)
org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:141)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
8/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
11.5% (57.3ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-06][write][T#7]'
2/10 snapshots sharing following 36 elements
org.elasticsearch.index.translog.TranslogWriter.add(TranslogWriter.java:160)
org.elasticsearch.index.translog.Translog.add(Translog.java:545)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:932)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:824)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:751)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:416)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:384)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:371)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:79)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:636)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:596)
org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2761)
org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2739)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$17(IndexShard.java:2699)
org.elasticsearch.index.shard.IndexShard$$Lambda$3426/2064690764.accept(Unknown Source)
org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:2806)
org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2698)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:1000)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:706)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:579)
org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:558)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:169)
com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:260)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:141)
com.floragunn.searchguard.SearchGuardPlugin$7$1.messageReceived(SearchGuardPlugin.java:634)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 8 elements
com.floragunn.searchguard.SearchGuardPlugin$7$1.messageReceived(SearchGuardPlugin.java:634)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1087)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
5/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {redacted-data-07}{2TSbcsZwTeyq2by30_GKjw}{eCvi7XZKTKKgfkj8Kni_vA}{x.y.48.126}{x.y.48.126:9300}{ml.machine_memory=67388059648, rack=us-central1-f, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:15.903Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
100.3% (501.3ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-07][[redacted-2019.09.19][0]: Lucene Merge Thread #35524]'
8/10 snapshots sharing following 12 elements
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
2/10 snapshots sharing following 7 elements
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
33.3% (166.4ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-07][write][T#2]'
6/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {redacted-data-01}{mbqlTDPTToekF5-xTz7Z9g}{M8j__xJiQC6o1tohgCRKfw}{x.y.63.213}{x.y.63.213:9300}{ml.machine_memory=67388059648, rack=us-central1-a, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
Hot threads at 2019-09-25T08:16:15.902Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
90.7% (453.6ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-01][[redacted-2019.09.19][1]: Lucene Merge Thread #32794]'
4/10 snapshots sharing following 14 elements
org.apache.lucene.codecs.compressing.LZ4.encodeLiterals(LZ4.java:155)
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:168)
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
3/10 snapshots sharing following 14 elements
org.apache.lucene.codecs.compressing.LZ4.encodeLiterals(LZ4.java:147)
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:168)
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
2/10 snapshots sharing following 12 elements
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
unique snapshot
java.nio.Bits.copyToArray(Bits.java:835)
java.nio.DirectByteBuffer.get(DirectByteBuffer.java:280)
org.apache.lucene.store.ByteBufferGuard.getBytes(ByteBufferGuard.java:93)
org.apache.lucene.store.ByteBufferIndexInput.readBytes(ByteBufferIndexInput.java:89)
org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:99)
org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.doReset(CompressingStoredFieldsReader.java:483)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.reset(CompressingStoredFieldsReader.java:389)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.document(CompressingStoredFieldsReader.java:568)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:641)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
90.7% (453.5ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-01][http_server_worker][T#13]'
4/10 snapshots sharing following 96 elements
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntry.getValue(Collections.java:1747)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.lambda$getResolvedIndexPattern$0(ConfigModel.java:646)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern$$Lambda$3051/1112143682.test(Unknown Source)
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.lambda$entryConsumer$0(Collections.java:1575)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$$Lambda$3042/1872138376.accept(Unknown Source)
java.util.TreeMap$EntrySpliterator.forEachRemaining(TreeMap.java:2969)
java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntrySetSpliterator.forEachRemaining(Collections.java:1600)
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.getResolvedIndexPattern(ConfigModel.java:647)
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$3992/545286910.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
6/10 snapshots sharing following 82 elements
com.floragunn.searchguard.sgconf.ConfigModel$IndexPattern.access$1100(ConfigModel.java:542)
com.floragunn.searchguard.sgconf.ConfigModel.impliesTypePerm(ConfigModel.java:844)
com.floragunn.searchguard.sgconf.ConfigModel.access$1300(ConfigModel.java:56)
com.floragunn.searchguard.sgconf.ConfigModel$SgRoles.impliesTypePermGlobal(ConfigModel.java:390)
com.floragunn.searchguard.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:614)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:242)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:107)
org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction.doExecute(TransportFieldCapabilitiesAction.java:45)
org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
com.floragunn.searchguard.filter.SearchGuardFilter.apply0(SearchGuardFilter.java:253)
com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:107)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
org.elasticsearch.client.support.AbstractClient.fieldCaps(AbstractClient.java:666)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction.lambda$prepareRequest$0(RestFieldCapabilitiesAction.java:84)
org.elasticsearch.rest.action.RestFieldCapabilitiesAction$$Lambda$3992/545286910.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:115)
com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:81)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:551)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1436)
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
java.lang.Thread.run(Thread.java:748)
::: {redacted-data-03}{2KdrOqE7QzSfjzXxFunNmw}{TaW7U8JDQE66oDyXSRVecw}{x.y.63.215}{x.y.63.215:9300}{ml.machine_memory=67388059648, rack=us-central1-b, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}
Hot threads at 2019-09-25T08:16:15.903Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
82.4% (411.9ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-03][[redacted-2019.09.19][1]: Lucene Merge Thread #69876]'
2/10 snapshots sharing following 8 elements
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:641)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
5/10 snapshots sharing following 16 elements
org.elasticsearch.common.lucene.store.FilterIndexOutput.writeBytes(FilterIndexOutput.java:59)
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73)
org.apache.lucene.codecs.compressing.LZ4.encodeLiterals(LZ4.java:155)
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:168)
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
2/10 snapshots sharing following 25 elements
java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
java.nio.channels.Channels.writeFully(Channels.java:101)
java.nio.channels.Channels.access$000(Channels.java:61)
java.nio.channels.Channels$1.write(Channels.java:174)
org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:417)
java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
java.io.BufferedOutputStream.write(BufferedOutputStream.java:95)
org.apache.lucene.store.OutputStreamIndexOutput.writeByte(OutputStreamIndexOutput.java:47)
org.elasticsearch.common.lucene.store.FilterIndexOutput.writeByte(FilterIndexOutput.java:54)
org.elasticsearch.index.store.ByteSizeCachingDirectory$2.writeByte(ByteSizeCachingDirectory.java:154)
org.apache.lucene.store.RateLimitedIndexOutput.writeByte(RateLimitedIndexOutput.java:66)
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:173)
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
unique snapshot
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:241)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:241)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:168)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:645)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:215)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:104)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
81.2% (406.1ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-03][write][T#7]'
2/10 snapshots sharing following 42 elements
java.lang.Object.notifyAll(Native Method)
org.elasticsearch.index.seqno.LocalCheckpointTracker.updateCheckpoint(LocalCheckpointTracker.java:209)
org.elasticsearch.index.seqno.LocalCheckpointTracker.markSeqNoAsCompleted(LocalCheckpointTracker.java:111)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:949)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:824)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:744)
org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:454)
org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$3923/805585702.get(Unknown Source)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:477)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:452)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1050)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1028)
org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:424)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:370)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction$$Lambda$3855/2014971952.accept(Unknown Source)
org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:987)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:369)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:324)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:311)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:169)
com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:139)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:91)
com.floragunn.searchguard.SearchGuardPlugin$7$1.messageReceived(SearchGuardPlugin.java:634)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
7/10 snapshots sharing following 46 elements
org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384)
org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485)
org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505)
org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:395)
org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384)
org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:96)
org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69)
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:281)
org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:799)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:775)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:744)
org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:454)
org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$3923/805585702.get(Unknown Source)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:477)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:452)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1050)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1028)
org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:424)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:370)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction$$Lambda$3855/2014971952.accept(Unknown Source)
org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:987)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:369)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:324)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:311)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:169)
com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:139)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:91)
com.floragunn.searchguard.SearchGuardPlugin$7$1.messageReceived(SearchGuardPlugin.java:634)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
unique snapshot
org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:417)
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394)
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251)
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1616)
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1164)
org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1109)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:924)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:824)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:744)
org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$3(TransportShardBulkAction.java:454)
org.elasticsearch.action.bulk.TransportShardBulkAction$$Lambda$3923/805585702.get(Unknown Source)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:477)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:452)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:216)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:159)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:151)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:139)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1050)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1028)
org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:424)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:370)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction$$Lambda$3855/2014971952.accept(Unknown Source)
org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240)
org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2561)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:987)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:369)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:324)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:311)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:169)
com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:139)
com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:91)
com.floragunn.searchguard.SearchGuardPlugin$7$1.messageReceived(SearchGuardPlugin.java:634)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
55.0% (274.7ms out of 500ms) cpu usage by thread 'elasticsearch[redacted-data-03][[.monitoring-es-6-2019.09.25][1]: Lucene Merge Thread #1852]'
4/10 snapshots sharing following 13 elements
org.apache.lucene.util.packed.MonotonicLongValues.get(MonotonicLongValues.java:40)
org.apache.lucene.util.packed.PackedLongValues.get(PackedLongValues.java:108)
org.apache.lucene.index.MergeState$1.get(MergeState.java:174)
org.apache.lucene.util.bkd.BKDWriter$MergeReader.next(BKDWriter.java:398)
org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:567)
org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:219)
org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:188)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
6/10 snapshots sharing following 8 elements
org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:219)
org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:188)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:137)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4482)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4077)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment