Skip to content

Instantly share code, notes, and snippets.

@vinicioflores

vinicioflores/hot threads 1

Created January 10, 2020 20:50
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save vinicioflores/fef2cd267224b1532c933eed57b4831c to your computer and use it in GitHub Desktop.
Save vinicioflores/fef2cd267224b1532c933eed57b4831c to your computer and use it in GitHub Desktop.
Download ZIP
Raw
hot threads 1
::: {arm-or-007_data}{9zEBQRPGTpSA_plraQfAMA}{ldXxb_vgT2ymHGKq8OoxBw}{plxcq8203.pdx.intel.com}{10.38.167.173:9300}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.407Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
5.3% (26.6ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-007_data][bulk][T#2]'
3/10 snapshots sharing following 32 elements
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:738)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:707)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:673)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:548)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeUpdateRequestOnce(TransportShardBulkAction.java:359)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeUpdateRequest(TransportShardBulkAction.java:406)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:240)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:123)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:110)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:72)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1034)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1012)
org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:103)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:359)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:299)
org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:975)
org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:972)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238)
org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2220)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:984)
org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:98)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:320)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:295)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:282)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:656)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
7/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
5.3% (26.4ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-007_data][bulk][T#1]'
2/10 snapshots sharing following 33 elements
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:452)
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:556)
org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:402)
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481)
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1729)
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1070)
org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1012)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:878)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:738)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:707)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:680)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:518)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:480)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:466)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:72)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:567)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:530)
org.elasticsearch.index.shard.IndexShard$2.onResponse(IndexShard.java:2314)
org.elasticsearch.index.shard.IndexShard$2.onResponse(IndexShard.java:2292)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238)
org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2291)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:641)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:513)
org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:493)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1555)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
8/10 snapshots sharing following 33 elements
org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:5030)
org.apache.lucene.index.DocumentsWriter$DeleteNewFilesEvent.process(DocumentsWriter.java:795)
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5106)
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5096)
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1732)
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1070)
org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1012)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:878)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:738)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:707)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:680)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:518)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:480)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:466)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:72)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:567)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:530)
org.elasticsearch.index.shard.IndexShard$2.onResponse(IndexShard.java:2314)
org.elasticsearch.index.shard.IndexShard$2.onResponse(IndexShard.java:2292)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238)
org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2291)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:641)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:513)
org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:493)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1555)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
5.0% (24.8ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-007_data][bulk][T#3]'
2/10 snapshots sharing following 5 elements
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {arm-or-002_master}{qm2d5LDBQey92KShf2M2-A}{5E2xkmZIRj2mzXFIPgOkQg}{plxcq8198.pdx.intel.com}{10.38.167.168:9301}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.377Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {arm-or-002_data}{WYL2OYuhQkGRzzsCD1XSUQ}{7pNJCaIURrK42LYSy1QWZA}{plxcq8198.pdx.intel.com}{10.38.167.168:9300}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.378Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
17.9% (89.3ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-002_data][[daas-log-prod-2020-01-ix][3]: Lucene Merge Thread #24631]'
2/10 snapshots sharing following 21 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.lucene.index.MergePolicy$OneMergeProgress.pauseNanos(MergePolicy.java:150)
org.apache.lucene.index.MergeRateLimiter.maybePause(MergeRateLimiter.java:148)
org.apache.lucene.index.MergeRateLimiter.pause(MergeRateLimiter.java:93)
org.apache.lucene.store.RateLimitedIndexOutput.checkRate(RateLimitedIndexOutput.java:78)
org.apache.lucene.store.RateLimitedIndexOutput.writeByte(RateLimitedIndexOutput.java:65)
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:176)
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:243)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:242)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:169)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:646)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:214)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:103)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
2/10 snapshots sharing following 23 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.lucene.index.MergePolicy$OneMergeProgress.pauseNanos(MergePolicy.java:150)
org.apache.lucene.index.MergeRateLimiter.maybePause(MergeRateLimiter.java:148)
org.apache.lucene.index.MergeRateLimiter.pause(MergeRateLimiter.java:93)
org.apache.lucene.store.RateLimitedIndexOutput.checkRate(RateLimitedIndexOutput.java:78)
org.apache.lucene.store.RateLimitedIndexOutput.writeByte(RateLimitedIndexOutput.java:65)
org.apache.lucene.codecs.compressing.LZ4.encodeLen(LZ4.java:145)
org.apache.lucene.codecs.compressing.LZ4.encodeLiterals(LZ4.java:153)
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:170)
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:243)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:242)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:169)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:646)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:214)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:103)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
5/10 snapshots sharing following 12 elements
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:243)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:242)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:169)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:646)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:214)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:103)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
unique snapshot
org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:135)
org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.doReset(CompressingStoredFieldsReader.java:483)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState.reset(CompressingStoredFieldsReader.java:389)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.document(CompressingStoredFieldsReader.java:568)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:642)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:214)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:103)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
::: {arm-or-006_data}{h7SOKT2mQAqkt91vHZg0rQ}{7kmok1r7QTKQwIGLfWVVkw}{plxcq8202.pdx.intel.com}{10.38.167.172:9300}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.383Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
4.1% (20.6ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-006_data][bulk][T#4]'
2/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {arm-or-005_master}{jydl9Gz2Shme0C9uSIAlDQ}{4ROV_GjcTFmJIBrWATWM-A}{plxcq8201.pdx.intel.com}{10.38.167.171:9301}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.376Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
0.0% (70.8micros out of 500ms) cpu usage by thread 'elasticsearch[arm-or-005_master][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
::: {arm-or-001_master}{vACvBzd5RiqvDDgWNvY6EQ}{Qrj_e2mRQWqNHff_blkr1Q}{plxcq8197.pdx.intel.com}{10.38.167.167:9301}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.376Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
0.0% (103.2micros out of 500ms) cpu usage by thread 'elasticsearch[arm-or-001_master][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
0.0% (59.5micros out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
10/10 snapshots sharing following 4 elements
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
java.lang.Thread.run(Thread.java:748)
::: {arm-or-003_master}{9WloaQB3RRS2keOlRB1hzA}{TzvxzzpTSOSnviEHlbdywA}{plxcq8199.pdx.intel.com}{10.38.167.169:9301}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.375Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
0.0% (121micros out of 500ms) cpu usage by thread 'elasticsearch[arm-or-003_master][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
::: {arm-or-008_data}{GGpw8eA5RpKJmLAC5dqyiQ}{qARuh-BYSCe9KMqsPr8g2Q}{plxcq8204.pdx.intel.com}{10.38.167.174:9300}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.401Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
34.5% (172.7ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-008_data][[daas-arm-prod-users-2020-01-ix][5]: Lucene Merge Thread #9489]'
7/10 snapshots sharing following 11 elements
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:230)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:115)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
3/10 snapshots sharing following 31 elements
sun.nio.ch.FileDispatcherImpl.write0(Native Method)
sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
sun.nio.ch.IOUtil.write(IOUtil.java:65)
sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
java.nio.channels.Channels.writeFully(Channels.java:101)
java.nio.channels.Channels.access$000(Channels.java:61)
java.nio.channels.Channels$1.write(Channels.java:174)
org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:419)
java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:53)
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73)
org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:52)
org.apache.lucene.store.RAMOutputStream.writeTo(RAMOutputStream.java:86)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlock(BlockTreeTermsWriter.java:830)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:624)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.pushTerm(BlockTreeTermsWriter.java:905)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:869)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:230)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:115)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
3.9% (19.3ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-008_data][bulk][T#1]'
4/10 snapshots sharing following 39 elements
sun.nio.ch.FileDispatcherImpl.write0(Native Method)
sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
sun.nio.ch.IOUtil.write(IOUtil.java:65)
sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
java.nio.channels.Channels.writeFully(Channels.java:101)
java.nio.channels.Channels.access$000(Channels.java:61)
java.nio.channels.Channels$1.write(Channels.java:174)
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
org.elasticsearch.common.bytes.BytesReference.writeTo(BytesReference.java:68)
org.elasticsearch.index.translog.TranslogWriter.add(TranslogWriter.java:195)
org.elasticsearch.index.translog.Translog.add(Translog.java:505)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:886)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:738)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:707)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:680)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:518)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:480)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:466)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:72)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:567)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:530)
org.elasticsearch.index.shard.IndexShard$2.onResponse(IndexShard.java:2314)
org.elasticsearch.index.shard.IndexShard$2.onResponse(IndexShard.java:2292)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238)
org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2291)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:641)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:513)
org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:493)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1555)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
6/10 snapshots sharing following 5 elements
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {arm-or-010_data}{_fIi48y9TUeeYMDgbqnEqA}{xLBZ2WSSQj6AhR2qLhhIBQ}{plxcq8206.pdx.intel.com}{10.38.167.176:9300}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.395Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
4.6% (22.8ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-010_data][bulk][T#1]'
9/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {arm-or-001_data}{qN88akkuQQ6F90isVQ6hxQ}{_EqIXxv4SiK-AcyyOZqpnA}{plxcq8197.pdx.intel.com}{10.38.167.167:9300}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.376Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
7.1% (35.2ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-001_data][bulk][T#3]'
7/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {arm-or-006_ingest}{DjAvKlWcRV-0imGYVyAhfQ}{BlEk6K0dTkidj1rjTJruIw}{plxcq8202.pdx.intel.com}{10.38.167.172:9302}{box_type=hot, ibi_site=pdx}
Hot threads at 2020-01-10T20:48:09.382Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {arm-or-008_ingest}{VAE3rT3gT-OABoQjPeo4BQ}{IAueAzU_T76ITx4dLVdo9g}{plxcq8204.pdx.intel.com}{10.38.167.174:9302}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.401Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
2.5% (12.2ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-008_ingest][http_server_worker][T#6]'
6/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
::: {arm-or-003_data}{R_GaKyzFRV6xTimYa6PPrg}{E0RFmGyPTOuRLkOyDURQ2w}{plxcq8199.pdx.intel.com}{10.38.167.169:9300}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.376Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
7.9% (39.2ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-003_data][bulk][T#3]'
7/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
7.6% (38.1ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-003_data][bulk][T#4]'
unique snapshot
org.apache.lucene.codecs.compressing.LZ4.encodeLiterals(LZ4.java:153)
org.apache.lucene.codecs.compressing.LZ4.encodeSequence(LZ4.java:170)
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:243)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:242)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:169)
org.apache.lucene.index.StoredFieldsConsumer.finishDocument(StoredFieldsConsumer.java:59)
org.apache.lucene.index.DefaultIndexingChain.finishStoredFields(DefaultIndexingChain.java:363)
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:403)
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:240)
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:496)
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1729)
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1070)
org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1012)
org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:878)
org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:738)
org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:707)
org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:673)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:548)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeUpdateRequestOnce(TransportShardBulkAction.java:359)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeUpdateRequest(TransportShardBulkAction.java:406)
org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:240)
org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:123)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:110)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:72)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1034)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1012)
org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:103)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:359)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:299)
org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:975)
org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:972)
org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238)
org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2220)
org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:984)
org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:98)
org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:320)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:295)
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:282)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:656)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
::: {arm-or-009_ingest}{a5voC7oXSf-78LfPlJ45Yg}{2pqZ9dJBTj2OuEyzWJvNCg}{plxcq8205.pdx.intel.com}{10.38.167.175:9302}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.492Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {arm-or-004_data}{qaQWdMVpToOq-18Ty2waaA}{zZjOUixCQG-4lSeRMw4XvQ}{plxcq8200.pdx.intel.com}{10.38.167.170:9300}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.376Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
35.6% (177.9ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-004_data][[discovery-prod-2020-01-10][3]: Lucene Merge Thread #5847]'
9/10 snapshots sharing following 12 elements
org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:243)
org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:242)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:169)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:646)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:214)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:103)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
unique snapshot
java.nio.Bits.copyToArray(Bits.java:835)
java.nio.DirectByteBuffer.get(DirectByteBuffer.java:280)
org.apache.lucene.store.ByteBufferGuard.getBytes(ByteBufferGuard.java:93)
org.apache.lucene.store.ByteBufferIndexInput.readBytes(ByteBufferIndexInput.java:89)
org.apache.lucene.store.BufferedChecksumIndexInput.readBytes(BufferedChecksumIndexInput.java:49)
org.apache.lucene.store.DataInput.readBytes(DataInput.java:87)
org.apache.lucene.store.DataInput.skipBytes(DataInput.java:317)
org.apache.lucene.store.ChecksumIndexInput.seek(ChecksumIndexInput.java:52)
org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:525)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.checkIntegrity(CompressingStoredFieldsReader.java:664)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:636)
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:214)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:103)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
::: {arm-or-005_data}{RqxttAceQFW1XC9tquQIWQ}{L5ap5XoXRsWF5W7Xq6SKaQ}{plxcq8201.pdx.intel.com}{10.38.167.171:9300}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.376Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: {arm-or-009_data}{f0aSRTkDQommICaE9nQRBg}{gO1PfI8qQiiF8P0xpeI7-A}{plxcq8205.pdx.intel.com}{10.38.167.175:9300}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.491Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
86.1% (430.6ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-009_data][management][T#2]'
10/10 snapshots sharing following 25 elements
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1448)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1437)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1426)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:375)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:953)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:106)
org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:50)
org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140)
org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262)
org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1555)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
25.4% (127ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-009_data][[discovery-prod-2020-01-10][9]: Lucene Merge Thread #5737]'
4/10 snapshots sharing following 13 elements
org.apache.lucene.index.MappingMultiPostingsEnum.nextDoc(MappingMultiPostingsEnum.java:103)
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:128)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:230)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:115)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
6/10 snapshots sharing following 11 elements
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:864)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:343)
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:230)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:115)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
::: {arm-or-004_master}{rzsh0R3LRxOeGeTp3BwTbA}{YuzXQ2skQxOxyIjKskxVkw}{plxcq8200.pdx.intel.com}{10.38.167.170:9301}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.377Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
0.0% (114.3micros out of 500ms) cpu usage by thread 'elasticsearch[arm-or-004_master][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
::: {arm-or-007_ingest}{fERzhejuRE6_MCMshvcArw}{RDh8CVPRQUWh21scNnVNtA}{plxcq8203.pdx.intel.com}{10.38.167.173:9302}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.377Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
2.3% (11.5ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-007_ingest][http_server_worker][T#4]'
7/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
::: {arm-or-010_ingest}{tGq_571FSkehKjbM_rHsUQ}{u4zYZLcOQqKLiLEE02so-w}{plxcq8206.pdx.intel.com}{10.38.167.176:9302}{ibi_site=pdx, box_type=hot}
Hot threads at 2020-01-10T20:48:09.395Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
8.0% (39.8ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-010_ingest][http_server_worker][T#1]'
5/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
2.0% (10ms out of 500ms) cpu usage by thread 'elasticsearch[arm-or-010_ingest][http_server_worker][T#4]'
4/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment