Skip to content

Instantly share code, notes, and snippets.

@EricSites
Last active December 20, 2015 15:09
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save EricSites/6152468 to your computer and use it in GitHub Desktop.
Save EricSites/6152468 to your computer and use it in GitHub Desktop.
I am getting java.lang.NullPointerException exception in my ElasticSearch cluster logs when I am doing a _bulk update or just an _update. I am sending a lot of data to my clusters. After I get this error I lose a shard and it has to be recreated.
curl -s -XPOST http://localhost:9200/samples/sample/_bulk --data-binary @requests; echo
[2013-08-04 21:48:54,972][WARN ][cluster.action.shard ] [es003] sending failed shard for [samples][20], node[COpFaS8rRhSulgDPAq5xxg], [R], s[STARTED], reason [Failed to perform [bulk/shard] on replica, message [RemoteTransportException; nested: ResponseHandlerFailureTransportException; nested: NullPointerException; ]]
[2013-08-04 21:48:55,028][WARN ][action.bulk ] [es003] Failed to perform bulk/shard on replica [samples][82]
org.elasticsearch.transport.RemoteTransportException
Caused by: org.elasticsearch.transport.ResponseHandlerFailureTransportException
Caused by: java.lang.NullPointerException
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:247)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:242)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$4.finishIfPossible(TransportShardReplicationOperationAction.java:693)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$4.handleResponse(TransportShardReplicationOperationAction.java:679)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$4.handleResponse(TransportShardReplicationOperationAction.java:676)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:153)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:124)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
[2013-08-04 21:48:55,029][WARN ][cluster.action.shard ] [es003] sending failed shard for [samples][82], node[aUq8CNUeT_iEIkJ7rVw02w], [R], s[STARTED], reason [Failed to perform [bulk/shard] on replica, message [RemoteTransportException; nested: ResponseHandlerFailureTransportException; nested: NullPointerException; ]]
{ "update" : { "_id" : "72707a88d84d11cd9d68b9d7748f16d0" } }
{"script" : "if (ctx._source['tracking'] != null) { if (ctx._source.tracking['vtest_batch_consumer_consumed_and_decoded'] != null) { ctx._source.tracking.vtest_batch_consumer_consumed_and_decoded += param1; } else { ctx._source.tracking['vtest_batch_consumer_consumed_and_decoded'] = 1; }} else { ctx._source.tracking = new_vtest_batch_consumer_consumed_and_decoded; }", "lang" : "js", "params" : {"param1" : 1, "new_vtest_batch_consumer_consumed_and_decoded" : {"vtest_batch_consumer_consumed_and_decoded" : 1 } },"upsert" : {"tracking" : {"vtest_batch_consumer_consumed_and_decoded" : 1 }}}
{"error":"ActionRequestValidationException[Validation Failed: 1: no requests added;]","status":500}
{ "update" : { "_id" : "06dbb83ee35c77fedb4676d3db9243d0" } }
{"script" : "if (ctx._source['tracking'] != null) { if (ctx._source.tracking['some_action'] != null) { ctx._source.tracking.some_action += param1; } else { ctx._source.tracking['some_action'] = 1; } } else { ctx._source.tracking = new_tracking; }","params" : {"param1" : 1, "new_tracking" : {"some_action" : 1 } }, "upsert" : {"tracking" : {"some_action" : 1 }}
}
{ "script" : "if (ctx._source['tracking'] != null) {
if (ctx._source.tracking['some_action'] != null) {
ctx._source.tracking.some_action += param1;
} else {
ctx._source.tracking['some_action'] = 1;
}
} else {
ctx._source.tracking = new_tracking;
}",
"params" : { "param1" : 1, "new_tracking" : { "some_action" : 1 } }, "upsert" : { "tracking" : { "some_action" : 1 } }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment