Elasticsearch Cluster

ElasticsearchCluster

Elasticsearch 1.1.0 with OpenJDK Java7 on Debian Wheezy

On 2 a.m. the cluster state switched to red


# head es_cluster.log
[2014-04-14 02:00:01,504][INFO ][cluster.metadata         ] [es@log01] [logstash-2014.04.14] creating index, cause [auto(bulk api)], shards [2]/[1], mappings [_default_]
[2014-04-14 02:00:02,938][INFO ][cluster.metadata         ] [es@log01] [puppetmaster-2014.04.14] creating index, cause [auto(bulk api)], shards [2]/[1], mappings []
[2014-04-14 10:46:12,318][INFO ][node                     ] [es@log01] stopping ...
[2014-04-14 10:46:12,446][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)


No log output at all!

I tried restarting and upgrading to 1.1.0 at 10 a.m.

Now I have several unassigned shards and I can't assign them...



# curl -s -XGET $(hostname -f):9200/_cat/health?v
epoch      timestamp cluster           status node.total node.data shards pri relo init unassign 
1397478614 14:30:14  es_cluster red             8         8   1803 910    0    0       43        
# curl -s -XGET $(hostname -f):9200/_cat/shards | grep -c UNASSIGNED
43
# curl -s -XGET $(hostname -f):9200/_cat/shards | grep UNASSIGNED | tail                                   
foobar                          1 p UNASSIGNED                                                             
foobar                          1 r UNASSIGNED                                                             
foobar                          2 p UNASSIGNED                                                             
foobar                          2 r UNASSIGNED                                                             
logstash-2014.01.28             1 r UNASSIGNED                                                             
logstash-2014.02.07             0 r UNASSIGNED                                                             
puppetmaster-2014.04.14-recover 0 p UNASSIGNED                                                             
puppetmaster-2014.04.14-recover 0 r UNASSIGNED                                                             
puppetmaster-2014.04.14-recover 1 p UNASSIGNED                                                             
puppetmaster-2014.04.14-recover 1 r UNASSIGNED                                                             
# curl -XPOST $(hostname -f):9200/_cluster/reroute?pretty=true -d '{"commands": [{"allocate": {"index": "foobar", "shard": 0, "node": "es@log09", "allow_primary": true }}]}' 
{
  "error" : "RemoteTransportException[[es@log05][inet[/12313.20.36.1337:9300]][cluster/reroute]]; nested: IllegalFormatConversionException[d != java.lang.Double]; ",
  "status" : 400
}

I accepted the data loss and deleted the unassigned shards...