Skip to content

Instantly share code, notes, and snippets.

@james1ewis

james1ewis/gist:4318255

Created December 17, 2012 13:24
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save james1ewis/4318255 to your computer and use it in GitHub Desktop.
Save james1ewis/4318255 to your computer and use it in GitHub Desktop.
Download ZIP
Elasticsearch log output showing the final gc warnings up to the first out of memory exception. This now repeats in this loop: a few gc warnings, an out of memory exception, a few gc warnings, an out of memory exception etc.
Raw
gistfile1.txt
[2012-12-17 13:19:46,198][WARN ][monitor.jvm ] [Autolycus] [gc][ConcurrentMarkSweep][6900][176] duration [20.8s], collections [1]/[21.4s], total [20.8s]/[41.6m], memory [5.9gb]->[5.9gb]/[5.9gb], all_pools {[Code Cache] [6.8mb]->[6.8mb]/[48mb]}{[Eden Space] [2.1kb]->[19.2kb]/[16.6mb]}{[Survivor Space] [0b]->[0b]/[2mb]}{[CMS Old Gen] [5.9gb]->[5.9gb]/[5.9gb]}{[CMS Perm Gen] [37.2mb]->[37.2mb]/[168mb]}
[2012-12-17 13:20:07,810][WARN ][monitor.jvm ] [Autolycus] [gc][ConcurrentMarkSweep][6901][177] duration [20.9s], collections [1]/[21.6s], total [20.9s]/[42m], memory [5.9gb]->[5.9gb]/[5.9gb], all_pools {[Code Cache] [6.8mb]->[6.8mb]/[48mb]}{[Eden Space] [19.2kb]->[2.1kb]/[16.6mb]}{[Survivor Space] [0b]->[0b]/[2mb]}{[CMS Old Gen] [5.9gb]->[5.9gb]/[5.9gb]}{[CMS Perm Gen] [37.2mb]->[37.2mb]/[168mb]}
[2012-12-17 13:20:41,924][WARN ][monitor.jvm ] [Autolycus] [gc][ConcurrentMarkSweep][6902][179] duration [33.7s], collections [2]/[34.1s], total [33.7s]/[42.5m], memory [5.9gb]->[5.9gb]/[5.9gb], all_pools {[Code Cache] [6.8mb]->[6.8mb]/[48mb]}{[Eden Space] [2.1kb]->[2kb]/[16.6mb]}{[Survivor Space] [0b]->[0b]/[2mb]}{[CMS Old Gen] [5.9gb]->[5.9gb]/[5.9gb]}{[CMS Perm Gen] [37.2mb]->[37.2mb]/[168mb]}
[2012-12-17 13:20:42,064][WARN ][http.netty ] [Autolycus] Caught exception while handling client http traffic, closing connection [id: 0xab62a47c, /10.0.10.217:60647 => /10.100.39.31:9200]
java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.common.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChannelBuffer.java:34)
at org.elasticsearch.common.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at org.elasticsearch.common.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:69)
at org.elasticsearch.common.netty.buffer.CompositeChannelBuffer.copy(CompositeChannelBuffer.java:568)
at org.elasticsearch.common.netty.buffer.AbstractChannelBuffer.copy(AbstractChannelBuffer.java:522)
at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.appendToCumulation(HttpChunkAggregator.java:207)
at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:174)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:793)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:455)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:538)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:437)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:793)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
[2012-12-17 13:21:12,918][WARN ][monitor.jvm ] [Autolycus] [gc][ConcurrentMarkSweep][6903][181] duration [30.6s], collections [2]/[30.9s], total [30.6s]/[43.1m], memory [5.9gb]->[5.9gb]/[5.9gb], all_pools {[Code Cache] [6.8mb]->[6.8mb]/[48mb]}{[Eden Space] [2kb]->[6.2kb]/[16.6mb]}{[Survivor Space] [0b]->[0b]/[2mb]}{[CMS Old Gen] [5.9gb]->[5.9gb]/[5.9gb]}{[CMS Perm Gen] [37.2mb]->[37.2mb]/[168mb]}
[2012-12-17 13:21:13,162][WARN ][http.netty ] [Autolycus] Caught exception while handling client http traffic, closing connection [id: 0xab62a47c, /10.0.10.217:60647 :> /10.100.39.31:9200]
java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.common.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42)
at org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChannelBuffer.java:34)
at org.elasticsearch.common.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at org.elasticsearch.common.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:69)
at org.elasticsearch.common.netty.buffer.CompositeChannelBuffer.copy(CompositeChannelBuffer.java:568)
at org.elasticsearch.common.netty.buffer.AbstractChannelBuffer.copy(AbstractChannelBuffer.java:522)
at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.appendToCumulation(HttpChunkAggregator.java:207)
at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:174)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:793)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:455)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:538)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:557)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:366)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:107)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:793)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:565)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at org.elasticsearch.common.netty.channel.Channels.fireChannelDisconnected(Channels.java:399)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:715)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:111)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:66)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:781)
at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:55)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:592)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
at org.elasticsearch.common.netty.channel.Channels.close(Channels.java:820)
at org.elasticsearch.common.netty.channel.AbstractChannel.close(AbstractChannel.java:197)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment