elasticsearch trying to allocate a primary shard which is disabled

es_pri_allocate_unassigned.md

Allow routing allocations:

curl -XPUT localhost:9200/_cluster/settings -d '{
                "transient" : {
                    "cluster.routing.allocation.enable" : "all"
                }
        }'

Reallocate unassigned shards, allowing primary allocations:

curl -s localhost:9200/_cat/shards | grep UNASS | while read line ; do \
  read -a fields <<<"$line" ;
  curl -XPOST -d '{ 
    "commands" : [ 
      { 
        "allocate" : {
          "index" : "'${fields[0]}'",
          "shard" : '${fields[1]}',
          "node" : "elasticsearch-'$(hostname)'",
          "allow_primary": "true"
        }
      }
    ]
  }' http://localhost:9200/_cluster/reroute?pretty ; done

In elastic 2.4.1 even after bringing the node up, elastic search didn't move the shards automagically. Have to use relocate api to move the unassigned nodes (due to node left) back to the node it self.

Just a heads up, I used this on a 21 node cluster (Elasticsearch 1.7) where one node had fallen over and after restarting the node there was an individual shard in 2 separate indexes that wouldn't allocate. I used this script to force the cluster to allocate the shards in an attempt to get the cluster out of RED, and unsurprisingly it "reset" the shard entirely and I lost 20 million docs or so.

There is no allocation explain API on 1.7 so I still don't know what I'll do if this occurs in the future.