Skip to content

Instantly share code, notes, and snippets.

@JorritSalverda

JorritSalverda/gist:eef328f1cf130f360409854394c321b6 Secret

Created February 7, 2017 14:34
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save JorritSalverda/eef328f1cf130f360409854394c321b6 to your computer and use it in GitHub Desktop.
Save JorritSalverda/eef328f1cf130f360409854394c321b6 to your computer and use it in GitHub Desktop.
Download ZIP
ElasticSearch 5.2.0 client node crash
Raw
gistfile1.txt
[2017-02-07T14:30:14,382][INFO ][o.e.n.Node ] [] initializing ...
[2017-02-07T14:30:14,498][INFO ][o.e.e.NodeEnvironment ] [dWtc6dR] using [1] data paths, mounts [[/data (/dev/sdb)]], net usable_space [9.1gb], net total_space [9.7gb], spins? [no], types [ext4]
[2017-02-07T14:30:14,499][INFO ][o.e.e.NodeEnvironment ] [dWtc6dR] heap size [5.9gb], compressed ordinary object pointers [true]
[2017-02-07T14:30:14,502][INFO ][o.e.n.Node ] node name [dWtc6dR] derived from node ID [dWtc6dRuTSiYL0CnOJn2ag]; set [node.name] to override
[2017-02-07T14:30:14,505][INFO ][o.e.n.Node ] version[5.2.0], pid[13], build[24e05b9/2017-01-24T19:52:35.800Z], OS[Linux/4.4.21+/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1
.8.0_111/25.111-b14]
[2017-02-07T14:30:16,818][INFO ][o.e.p.p.PrometheusExporterPlugin] starting Prometheus exporter plugin...
[2017-02-07T14:30:16,824][INFO ][o.e.p.PluginsService ] [dWtc6dR] loaded module [aggs-matrix-stats]
[2017-02-07T14:30:16,825][INFO ][o.e.p.PluginsService ] [dWtc6dR] loaded module [ingest-common]
[2017-02-07T14:30:16,825][INFO ][o.e.p.PluginsService ] [dWtc6dR] loaded module [lang-expression]
[2017-02-07T14:30:16,825][INFO ][o.e.p.PluginsService ] [dWtc6dR] loaded module [lang-groovy]
[2017-02-07T14:30:16,825][INFO ][o.e.p.PluginsService ] [dWtc6dR] loaded module [lang-mustache]
[2017-02-07T14:30:16,825][INFO ][o.e.p.PluginsService ] [dWtc6dR] loaded module [lang-painless]
[2017-02-07T14:30:16,825][INFO ][o.e.p.PluginsService ] [dWtc6dR] loaded module [percolator]
[2017-02-07T14:30:16,825][INFO ][o.e.p.PluginsService ] [dWtc6dR] loaded module [reindex]
[2017-02-07T14:30:16,826][INFO ][o.e.p.PluginsService ] [dWtc6dR] loaded module [transport-netty3]
[2017-02-07T14:30:16,826][INFO ][o.e.p.PluginsService ] [dWtc6dR] loaded module [transport-netty4]
[2017-02-07T14:30:16,828][INFO ][o.e.p.PluginsService ] [dWtc6dR] loaded plugin [prometheus-exporter]
[2017-02-07T14:30:17,275][WARN ][o.e.d.s.g.GroovyScriptEngineService] [groovy] scripts are deprecated, use [painless] scripts instead
[2017-02-07T14:30:25,947][INFO ][o.e.n.Node ] initialized
[2017-02-07T14:30:25,947][INFO ][o.e.n.Node ] [dWtc6dR] starting ...
[2017-02-07T14:30:26,272][WARN ][i.n.u.i.MacAddressUtil ] Failed to find a usable hardware address from the network interfaces; using random bytes: 37:06:3e:02:d4:be:9d:f3
[2017-02-07T14:30:26,539][INFO ][o.e.t.TransportService ] [dWtc6dR] publish_address {10.120.54.112:9300}, bound_addresses {10.120.54.112:9300}
[2017-02-07T14:30:26,552][INFO ][o.e.b.BootstrapChecks ] [dWtc6dR] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-02-07T14:30:34,655][INFO ][o.e.c.s.ClusterService ] [dWtc6dR] detected_master {-lbfIdE}{-lbfIdEuSlmrizlZnIEPdA}{2ZEQMX0qQduXqEFibLmQ5w}{10.120.22.38}{10.120.22.38:9300}, added {{H_3vmgi}{H_3vm
giXRrqDG3tso4QzqA}{tGhYbY3MRDWyhsYMN5Eccg}{10.120.40.5}{10.120.40.5:9300},{mL673DR}{mL673DRXR1eUzwyNTFXVsg}{oJdzHJuNSmanyMZJ3-gJkQ}{10.120.43.5}{10.120.43.5:9300},{Oux2bIR}{Oux2bIRcSTCPBpr5vimhRA}{t5z
vhGEZThugOoBHwPV01A}{10.120.56.9}{10.120.56.9:9300},{-lbfIdE}{-lbfIdEuSlmrizlZnIEPdA}{2ZEQMX0qQduXqEFibLmQ5w}{10.120.22.38}{10.120.22.38:9300},{8sJMfrA}{8sJMfrA9SEWZZtIkHcngAg}{awvAqYP9Q6-PJrtsfRFHSg}
{10.120.31.13}{10.120.31.13:9300},{jP9BNBs}{jP9BNBsvTziSKkobJckYLQ}{3PO9ZGdIRrynXjGfOD_Mng}{10.120.56.15}{10.120.56.15:9300},{iKjx7jL}{iKjx7jLsTi-zzxqiEgS3Ew}{5h018mBpTrGpvJ07KoJGpA}{10.120.18.37}{10.
120.18.37:9300},{RiGPF0z}{RiGPF0zjQu-w1tMGOlgHzA}{tJ-e42fASR2xblMCN5KXCw}{10.120.41.6}{10.120.41.6:9300},{LAwIUXi}{LAwIUXiHSqqvWwSgT4wJuQ}{h-j82NURTnqLEIhrZRY5bw}{10.120.44.5}{10.120.44.5:9300},{IBgrv
hi}{IBgrvhioSlehxxwg3retjg}{6RR1vVAcSV2XCXj4TVbMgA}{10.120.53.26}{10.120.53.26:9300},{lRtuQpw}{lRtuQpwuTGujSeUVRc07_w}{OW0JwPU8Qmy4Il2OjoL-wg}{10.120.22.18}{10.120.22.18:9300},{c1_8yH-}{c1_8yH-CQl2Mjm
rbpjPOag}{eIHamX5BREKHRWBdO9EXhw}{10.120.22.39}{10.120.22.39:9300},{e2Q4KZJ}{e2Q4KZJySKSF0HC96xML1g}{docVyjQ-RUqahLphTEbbDA}{10.120.39.5}{10.120.39.5:9300},}, reason: zen-disco-receive(from master [ma
ster {-lbfIdE}{-lbfIdEuSlmrizlZnIEPdA}{2ZEQMX0qQduXqEFibLmQ5w}{10.120.22.38}{10.120.22.38:9300} committed version [3449]])
[2017-02-07T14:30:34,841][INFO ][o.e.c.s.ClusterSettings ] [dWtc6dR] updating [cluster.routing.allocation.exclude.] from [{}] to [{"_ip":"10.120.194.17,10.120.22.18,10.120.31.13,10.120.6.20,10.120.6.
4"}]
[2017-02-07T14:30:34,873][INFO ][o.e.h.HttpServer ] [dWtc6dR] publish_address {10.120.54.112:9200}, bound_addresses {10.120.54.112:9200}
[2017-02-07T14:30:34,873][INFO ][o.e.n.Node ] [dWtc6dR] started
[2017-02-07T14:30:38,609][INFO ][o.e.c.s.ClusterService ] [dWtc6dR] added {{Rzx73Y8}{Rzx73Y8URmagwMzwMhNR_Q}{MsiNlqXaS6aushDPnjBOyg}{10.120.57.5}{10.120.57.5:9300},}, reason: zen-disco-receive(from
master [master {-lbfIdE}{-lbfIdEuSlmrizlZnIEPdA}{2ZEQMX0qQduXqEFibLmQ5w}{10.120.22.38}{10.120.22.38:9300} committed version [3450]])
[2017-02-07T14:30:53,193][INFO ][o.e.c.s.ClusterService ] [dWtc6dR] added {{Fvd0jHi}{Fvd0jHiHTauihGtGC3Y7Jw}{3oY_Gnr0SJyvHmEpj81iTw}{10.120.57.10}{10.120.57.10:9300},}, reason: zen-disco-receive(fro
m master [master {-lbfIdE}{-lbfIdEuSlmrizlZnIEPdA}{2ZEQMX0qQduXqEFibLmQ5w}{10.120.22.38}{10.120.22.38:9300} committed version [3451]])
[2017-02-07T14:31:25,912][INFO ][o.e.c.s.ClusterService ] [dWtc6dR] removed {{Rzx73Y8}{Rzx73Y8URmagwMzwMhNR_Q}{MsiNlqXaS6aushDPnjBOyg}{10.120.57.5}{10.120.57.5:9300},}, reason: zen-disco-receive(fro
m master [master {-lbfIdE}{-lbfIdEuSlmrizlZnIEPdA}{2ZEQMX0qQduXqEFibLmQ5w}{10.120.22.38}{10.120.22.38:9300} committed version [3452]])
[2017-02-07T14:31:26,170][INFO ][o.e.c.s.ClusterService ] [dWtc6dR] removed {{Fvd0jHi}{Fvd0jHiHTauihGtGC3Y7Jw}{3oY_Gnr0SJyvHmEpj81iTw}{10.120.57.10}{10.120.57.10:9300},}, reason: zen-disco-receive(f
rom master [master {-lbfIdE}{-lbfIdEuSlmrizlZnIEPdA}{2ZEQMX0qQduXqEFibLmQ5w}{10.120.22.38}{10.120.22.38:9300} committed version [3453]])
[2017-02-07T14:31:26,188][WARN ][o.e.c.NodeConnectionsService] [dWtc6dR] failed to connect to node {Fvd0jHi}{Fvd0jHiHTauihGtGC3Y7Jw}{3oY_Gnr0SJyvHmEpj81iTw}{10.120.57.10}{10.120.57.10:9300} (tried [1]
times)
org.elasticsearch.transport.ConnectTransportException: [Fvd0jHi][10.120.57.10:9300] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:370) ~[?:?]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:495) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:460) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:318) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:305) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.cluster.NodeConnectionsService.validateNodeConnected(NodeConnectionsService.java:121) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.cluster.NodeConnectionsService$ConnectionChecker.doRun(NodeConnectionsService.java:150) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.0.jar:5.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.120.57.10/10.120.57.10:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:346) ~[?:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:630) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
... 1 more
[2017-02-07T14:31:26,252][WARN ][o.e.a.a.c.n.i.TransportNodesInfoAction] [dWtc6dR] not accumulating exceptions, excluding exception from response
org.elasticsearch.action.FailedNodeException: Failed node [Fvd0jHiHTauihGtGC3Y7Jw]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:247) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$300(TransportNodesAction.java:160) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:219) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:476) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:205) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:145) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:87) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:75) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:64) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:710) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesInfo(AbstractClient.java:791) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.rest.action.admin.cluster.RestNodesInfoAction.lambda$prepareRequest$0(RestNodesInfoAction.java:111) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:82) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:162) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.http.HttpServer.dispatchRequest(HttpServer.java:115) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:515) ~[transport-netty4-5.2.0.jar:5.2.0]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:70) ~[transport-netty4-5.2.0.jar:5.2.0]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:66) ~[transport-netty4-5.2.0.jar:5.2.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at org.elasticsearch.http.netty4.cors.Netty4CorsHandler.channelRead(Netty4CorsHandler.java:76) ~[transport-netty4-5.2.0.jar:5.2.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.7.Final.jar:4.1.7.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: org.elasticsearch.transport.NodeNotConnectedException: [Fvd0jHi][10.120.57.10:9300] Node not connected
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:590) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:113) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:496) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:472) ~[elasticsearch-5.2.0.jar:5.2.0]
... 62 more
[2017-02-07T14:31:49,533][INFO ][o.e.m.j.JvmGcMonitorService] [dWtc6dR] [gc][83] overhead, spent [251ms] collecting in the last [1s]
[2017-02-07T14:31:53,963][WARN ][o.e.m.j.JvmGcMonitorService] [dWtc6dR] [gc][86] overhead, spent [1.9s] collecting in the last [2.3s]
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid13.hprof ...
Unable to create java_pid13.hprof: Permission denied
[2017-02-07T14:31:58,360][ERROR][o.e.t.n.Netty4Utils ] fatal error on the network layer
at org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:140)
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:77)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:78)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1301)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:914)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:99)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:140)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:745)
[2017-02-07T14:31:58,359][WARN ][o.e.m.j.JvmGcMonitorService] [dWtc6dR] [gc][87] overhead, spent [4.3s] collecting in the last [4.3s]
[2017-02-07T14:32:02,102][ERROR][o.e.t.n.Netty4Utils ] fatal error on the network layer
at org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:140)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:83)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257)
at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1301)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265)
at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:914)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:99)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:140)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:745)
[2017-02-07T14:32:02,104][WARN ][o.e.m.j.JvmGcMonitorService] [dWtc6dR] [gc][88] overhead, spent [3.6s] collecting in the last [3.7s]
[2017-02-07T14:32:04,384][WARN ][o.e.m.j.JvmGcMonitorService] [dWtc6dR] [gc][89] overhead, spent [2.1s] collecting in the last [2.2s]
[2017-02-07T14:32:04,394][WARN ][o.e.h.n.Netty4HttpServerTransport] [dWtc6dR] caught exception while handling client http traffic, closing connection [id: 0x35a57eb8, L:/10.120.54.112:9200 - R:/10.120
.39.2:52328]
java.lang.OutOfMemoryError: Java heap space
at io.netty.buffer.PoolArena$HeapArena.newChunk(PoolArena.java:656) ~[netty-buffer-4.1.7.Final.jar:4.1.7.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237) ~[netty-buffer-4.1.7.Final.jar:4.1.7.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:221) ~[netty-buffer-4.1.7.Final.jar:4.1.7.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:141) ~[netty-buffer-4.1.7.Final.jar:4.1.7.Final]
at io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:272) ~[netty-buffer-4.1.7.Final.jar:4.1.7.Final]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:160) ~[netty-buffer-4.1.7.Final.jar:4.1.7.Final]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:151) ~[netty-buffer-4.1.7.Final.jar:4.1.7.Final]
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:133) ~[netty-buffer-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:73) ~[netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.7.Final.jar:4.1.7.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
[2017-02-07T14:31:58,415][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [Thread-5], exiting
java.lang.OutOfMemoryError: Java heap space
at io.netty.buffer.PoolArena$HeapArena.newChunk(PoolArena.java:656) ~[?:?]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237) ~[?:?]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:221) ~[?:?]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:141) ~[?:?]
at io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:272) ~[?:?]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:160) ~[?:?]
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:151) ~[?:?]
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:133) ~[?:?]
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:73) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
[2017-02-07T14:32:06,234][WARN ][o.e.m.j.JvmGcMonitorService] [dWtc6dR] [gc][90] overhead, spent [1.8s] collecting in the last [1.8s]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment