Skip to content

Instantly share code, notes, and snippets.

@takism1

takism1/es-ny01-20130805 Secret

Created August 5, 2013 20:59
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save takism1/312e2bf2d0ff583a235f to your computer and use it in GitHub Desktop.
Save takism1/312e2bf2d0ff583a235f to your computer and use it in GitHub Desktop.
Download ZIP
Output from ny01 node
Raw
es-ny01-20130805
[2013-08-04 07:00:09,395][DEBUG][cluster.service ] [ny01] processing [zen-disco-receive(join from node[[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}])]: execute
[2013-08-04 07:00:09,395][WARN ][discovery.zen ] [ny01] received a join request for an existing node [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}]
[2013-08-04 07:00:09,396][TRACE][cluster.service ] [ny01] cluster state updated:
version [42], source [zen-disco-receive(join from node[[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}])]
nodes:
[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}
[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}, local, master
routing_table:
-- index [my_index]
----shard_id [my_index][0]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][1]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][2]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][3]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
routing_nodes:
-----node_id[mhhvyRRoQ6WQX52Ov-GkjQ][V]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
-----node_id[NRPvTu1yToiC7SKOxPxNeA][V]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
-----node_id[fy3VCsuOSm-eqI35T4afSg][V]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
-----node_id[HPRnuckbQg2-r-Hf9jBXcg][V]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
---- unassigned
[2013-08-04 07:00:09,397][DEBUG][cluster.service ] [ny01] processing [zen-disco-receive(join from node[[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}])]: done applying updated cluster_state
[2013-08-04 07:00:09,450][WARN ][cluster.action.shard ] [ny01] received shard failed for [my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,452][WARN ][cluster.action.shard ] [ny01] received shard failed for [my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,452][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: execute
[2013-08-04 07:00:09,452][WARN ][cluster.action.shard ] [ny01] received shard failed for [my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,452][DEBUG][cluster.action.shard ] [ny01] Received failed shard [my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,452][WARN ][cluster.action.shard ] [ny01] received shard failed for [my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,453][TRACE][cluster.routing.allocation.allocator] [ny01] Start balancing cluster
[2013-08-04 07:00:09,453][TRACE][cluster.routing.allocation.allocator] [ny01] Start distributing Shards
[2013-08-04 07:00:09,453][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,453][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,453][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,453][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,453][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,453][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,453][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,453][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,453][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,453][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,454][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,454][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,454][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,454][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,454][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,454][TRACE][cluster.routing.allocation.allocator] [ny01] Start allocating unassigned shards
[2013-08-04 07:00:09,454][TRACE][cluster.routing.allocation.allocator] [ny01] No Node found to assign shard [[my_index][0], node[null], [R], s[UNASSIGNED]]
[2013-08-04 07:00:09,454][TRACE][cluster.routing.allocation.allocator] [ny01] Stop balancing index [my_index] min_node [NRPvTu1yToiC7SKOxPxNeA] weight: [0.15] max_node [fy3VCsuOSm-eqI35T4afSg] weight: [-1.0] delta: [1.0]
[2013-08-04 07:00:09,454][TRACE][cluster.routing.allocation.allocator] [ny01] Start balancing cluster
[2013-08-04 07:00:09,454][TRACE][cluster.routing.allocation.allocator] [ny01] Start distributing Shards
[2013-08-04 07:00:09,454][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,454][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,454][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Start allocating unassigned shards
[2013-08-04 07:00:09,455][TRACE][cluster.routing.allocation.allocator] [ny01] Stop balancing index [my_index] min_node [NRPvTu1yToiC7SKOxPxNeA] weight: [0.15] max_node [fy3VCsuOSm-eqI35T4afSg] weight: [-1.0] delta: [1.0]
[2013-08-04 07:00:09,456][DEBUG][cluster.action.shard ] [ny01] Applying failed shard [my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,456][TRACE][cluster.service ] [ny01] cluster state updated:
version [43], source [shard-failed ([my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]
nodes:
[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}
[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}, local, master
routing_table:
-- index [my_index]
----shard_id [my_index][0]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][0], node[null], [R], s[UNASSIGNED]
----shard_id [my_index][1]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][2]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][3]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
routing_nodes:
-----node_id[mhhvyRRoQ6WQX52Ov-GkjQ][V]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
-----node_id[NRPvTu1yToiC7SKOxPxNeA][V]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
-----node_id[fy3VCsuOSm-eqI35T4afSg][V]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
-----node_id[HPRnuckbQg2-r-Hf9jBXcg][V]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
---- unassigned
--------[my_index][0], node[null], [R], s[UNASSIGNED]
[2013-08-04 07:00:09,457][TRACE][gateway.local.state.shards] [ny01] [my_index][0] writing shard state, reason [version changed from [56] to [57]]
[2013-08-04 07:00:09,459][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: done applying updated cluster_state
[2013-08-04 07:00:09,459][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: execute
[2013-08-04 07:00:09,459][DEBUG][cluster.action.shard ] [ny01] Received failed shard [my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,472][TRACE][gateway.local ] [ny01] [my_index][0], node[null], [R], s[UNASSIGNED]: checking node [[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}]
[2013-08-04 07:00:09,473][TRACE][gateway.local ] [ny01] [my_index][0], node[null], [R], s[UNASSIGNED]: checking node [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}]
[2013-08-04 07:00:09,473][TRACE][gateway.local ] [ny01] [my_index][0], node[null], [R], s[UNASSIGNED]: checking node [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}]
[2013-08-04 07:00:09,473][TRACE][gateway.local ] [ny01] [my_index][0], node[null], [R], s[UNASSIGNED]: checking node [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}]
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Start balancing cluster
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Start distributing Shards
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,473][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,474][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,474][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,474][TRACE][cluster.routing.allocation.allocator] [ny01] Start allocating unassigned shards
[2013-08-04 07:00:09,474][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[null], [R], s[UNASSIGNED]] to [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,474][TRACE][cluster.routing.allocation.allocator] [ny01] No Node found to assign shard [[my_index][3], node[null], [R], s[UNASSIGNED]]
[2013-08-04 07:00:09,474][TRACE][cluster.routing.allocation.allocator] [ny01] Stop balancing index [my_index] min_node [NRPvTu1yToiC7SKOxPxNeA] weight: [0.15] max_node [fy3VCsuOSm-eqI35T4afSg] weight: [-1.0] delta: [1.0]
[2013-08-04 07:00:09,474][TRACE][cluster.routing.allocation.allocator] [ny01] Start balancing cluster
[2013-08-04 07:00:09,474][TRACE][cluster.routing.allocation.allocator] [ny01] Start distributing Shards
[2013-08-04 07:00:09,474][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,474][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,474][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,474][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,475][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,475][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,475][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,475][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,475][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,475][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,475][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,475][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,475][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,475][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,475][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,475][TRACE][cluster.routing.allocation.allocator] [ny01] Start allocating unassigned shards
[2013-08-04 07:00:09,475][TRACE][cluster.routing.allocation.allocator] [ny01] Stop balancing index [my_index] min_node [NRPvTu1yToiC7SKOxPxNeA] weight: [0.15] max_node [fy3VCsuOSm-eqI35T4afSg] weight: [-1.0] delta: [1.0]
[2013-08-04 07:00:09,475][DEBUG][cluster.action.shard ] [ny01] Applying failed shard [my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,476][TRACE][cluster.service ] [ny01] cluster state updated:
version [44], source [shard-failed ([my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]
nodes:
[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}
[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}, local, master
routing_table:
-- index [my_index]
----shard_id [my_index][0]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][1]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][2]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][3]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][3], node[null], [R], s[UNASSIGNED]
routing_nodes:
-----node_id[mhhvyRRoQ6WQX52Ov-GkjQ][V]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
-----node_id[NRPvTu1yToiC7SKOxPxNeA][V]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
-----node_id[fy3VCsuOSm-eqI35T4afSg][V]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
-----node_id[HPRnuckbQg2-r-Hf9jBXcg][V]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
---- unassigned
--------[my_index][3], node[null], [R], s[UNASSIGNED]
[2013-08-04 07:00:09,476][TRACE][gateway.local.state.shards] [ny01] [my_index][3] writing shard state, reason [version changed from [59] to [60]]
[2013-08-04 07:00:09,478][TRACE][gateway.local.state.shards] [ny01] [my_index][0] writing shard state, reason [version changed from [57] to [58]]
[2013-08-04 07:00:09,480][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: done applying updated cluster_state
[2013-08-04 07:00:09,480][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: execute
[2013-08-04 07:00:09,480][DEBUG][cluster.action.shard ] [ny01] Received failed shard [my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,485][TRACE][gateway.local ] [ny01] [my_index][3], node[null], [R], s[UNASSIGNED]: checking node [[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}]
[2013-08-04 07:00:09,485][TRACE][gateway.local ] [ny01] [my_index][3], node[null], [R], s[UNASSIGNED]: checking node [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}]
[2013-08-04 07:00:09,485][TRACE][gateway.local ] [ny01] [my_index][3], node[null], [R], s[UNASSIGNED]: checking node [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}]
[2013-08-04 07:00:09,486][TRACE][gateway.local ] [ny01] [my_index][3], node[null], [R], s[UNASSIGNED]: checking node [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Start balancing cluster
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Start distributing Shards
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,486][TRACE][cluster.routing.allocation.allocator] [ny01] Start allocating unassigned shards
[2013-08-04 07:00:09,487][TRACE][cluster.routing.allocation.allocator] [ny01] No Node found to assign shard [[my_index][1], node[null], [R], s[UNASSIGNED]]
[2013-08-04 07:00:09,487][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[null], [R], s[UNASSIGNED]] to [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,487][TRACE][cluster.routing.allocation.allocator] [ny01] Stop balancing index [my_index] min_node [NRPvTu1yToiC7SKOxPxNeA] weight: [0.15] max_node [fy3VCsuOSm-eqI35T4afSg] weight: [-1.0] delta: [1.0]
[2013-08-04 07:00:09,487][TRACE][cluster.routing.allocation.allocator] [ny01] Start balancing cluster
[2013-08-04 07:00:09,487][TRACE][cluster.routing.allocation.allocator] [ny01] Start distributing Shards
[2013-08-04 07:00:09,487][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,487][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,487][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,487][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,487][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,487][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,487][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,487][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,488][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,488][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,488][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,488][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,488][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,488][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,488][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,488][TRACE][cluster.routing.allocation.allocator] [ny01] Start allocating unassigned shards
[2013-08-04 07:00:09,488][TRACE][cluster.routing.allocation.allocator] [ny01] Stop balancing index [my_index] min_node [NRPvTu1yToiC7SKOxPxNeA] weight: [0.15] max_node [fy3VCsuOSm-eqI35T4afSg] weight: [-1.0] delta: [1.0]
[2013-08-04 07:00:09,488][DEBUG][cluster.action.shard ] [ny01] Applying failed shard [my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,488][TRACE][cluster.service ] [ny01] cluster state updated:
version [45], source [shard-failed ([my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]
nodes:
[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}
[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}, local, master
routing_table:
-- index [my_index]
----shard_id [my_index][0]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][1]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[null], [R], s[UNASSIGNED]
----shard_id [my_index][2]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][3]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
routing_nodes:
-----node_id[mhhvyRRoQ6WQX52Ov-GkjQ][V]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
-----node_id[NRPvTu1yToiC7SKOxPxNeA][V]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
-----node_id[fy3VCsuOSm-eqI35T4afSg][V]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
-----node_id[HPRnuckbQg2-r-Hf9jBXcg][V]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
---- unassigned
--------[my_index][1], node[null], [R], s[UNASSIGNED]
[2013-08-04 07:00:09,489][TRACE][gateway.local.state.shards] [ny01] [my_index][3] writing shard state, reason [version changed from [60] to [61]]
[2013-08-04 07:00:09,491][TRACE][gateway.local.state.shards] [ny01] [my_index][1] writing shard state, reason [version changed from [57] to [58]]
[2013-08-04 07:00:09,493][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: done applying updated cluster_state
[2013-08-04 07:00:09,493][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: execute
[2013-08-04 07:00:09,493][DEBUG][cluster.action.shard ] [ny01] Received failed shard [my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,493][TRACE][cluster.routing.allocation.allocator] [ny01] Start balancing cluster
[2013-08-04 07:00:09,493][TRACE][cluster.routing.allocation.allocator] [ny01] Start distributing Shards
[2013-08-04 07:00:09,493][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,493][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,493][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,493][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,493][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,493][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,493][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,493][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Start allocating unassigned shards
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] No Node found to assign shard [[my_index][2], node[null], [R], s[UNASSIGNED]]
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Stop balancing index [my_index] min_node [NRPvTu1yToiC7SKOxPxNeA] weight: [0.15] max_node [fy3VCsuOSm-eqI35T4afSg] weight: [-1.0] delta: [1.0]
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Start balancing cluster
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Start distributing Shards
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,494][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]] to node [mhhvyRRoQ6WQX52Ov-GkjQ]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]] to node [NRPvTu1yToiC7SKOxPxNeA]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Assigned shard [[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]] to node [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Start allocating unassigned shards
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Balancing from node [NRPvTu1yToiC7SKOxPxNeA] weight: [0.15] to node [fy3VCsuOSm-eqI35T4afSg] weight: [-1.9499999] delta: [2.0]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Try relocating shard for index index [my_index] from node [NRPvTu1yToiC7SKOxPxNeA] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Couldn't find shard to relocate from node [NRPvTu1yToiC7SKOxPxNeA] to node [fy3VCsuOSm-eqI35T4afSg]
[2013-08-04 07:00:09,495][TRACE][cluster.routing.allocation.allocator] [ny01] Stop balancing index [my_index] min_node [NRPvTu1yToiC7SKOxPxNeA] weight: [0.15] max_node [mhhvyRRoQ6WQX52Ov-GkjQ] weight: [-0.05] delta: [0.2]
[2013-08-04 07:00:09,496][DEBUG][cluster.action.shard ] [ny01] Applying failed shard [my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,496][TRACE][cluster.service ] [ny01] cluster state updated:
version [46], source [shard-failed ([my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]
nodes:
[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}
[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}, local, master
routing_table:
-- index [my_index]
----shard_id [my_index][0]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][1]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[null], [R], s[UNASSIGNED]
----shard_id [my_index][2]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][2], node[null], [R], s[UNASSIGNED]
----shard_id [my_index][3]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
routing_nodes:
-----node_id[mhhvyRRoQ6WQX52Ov-GkjQ][V]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
-----node_id[NRPvTu1yToiC7SKOxPxNeA][V]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
-----node_id[fy3VCsuOSm-eqI35T4afSg][V]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
-----node_id[HPRnuckbQg2-r-Hf9jBXcg][V]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
---- unassigned
--------[my_index][1], node[null], [R], s[UNASSIGNED]
--------[my_index][2], node[null], [R], s[UNASSIGNED]
[2013-08-04 07:00:09,496][TRACE][gateway.local.state.shards] [ny01] [my_index][2] writing shard state, reason [version changed from [58] to [59]]
[2013-08-04 07:00:09,496][WARN ][cluster.action.shard ] [ny01] received shard failed for [my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,497][WARN ][cluster.action.shard ] [ny01] received shard failed for [my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,497][WARN ][cluster.action.shard ] [ny01] received shard failed for [my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,498][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: done applying updated cluster_state
[2013-08-04 07:00:09,499][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: execute
[2013-08-04 07:00:09,499][DEBUG][cluster.action.shard ] [ny01] Received failed shard [my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,499][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: no change in cluster_state
[2013-08-04 07:00:09,499][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: execute
[2013-08-04 07:00:09,499][DEBUG][cluster.action.shard ] [ny01] Received failed shard [my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,499][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: no change in cluster_state
[2013-08-04 07:00:09,499][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: execute
[2013-08-04 07:00:09,499][DEBUG][cluster.action.shard ] [ny01] Received failed shard [my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,499][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: no change in cluster_state
[2013-08-04 07:00:09,563][WARN ][cluster.action.shard ] [ny01] received shard failed for [my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,564][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: execute
[2013-08-04 07:00:09,564][WARN ][cluster.action.shard ] [ny01] received shard failed for [my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,564][DEBUG][cluster.action.shard ] [ny01] Received failed shard [my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,564][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: no change in cluster_state
[2013-08-04 07:00:09,564][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: execute
[2013-08-04 07:00:09,564][DEBUG][cluster.action.shard ] [ny01] Received failed shard [my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,564][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: no change in cluster_state
[2013-08-04 07:00:09,565][WARN ][cluster.action.shard ] [ny01] received shard failed for [my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,565][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: execute
[2013-08-04 07:00:09,565][DEBUG][cluster.action.shard ] [ny01] Received failed shard [my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,565][DEBUG][cluster.service ] [ny01] processing [shard-failed ([my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]), reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]]: no change in cluster_state
[2013-08-04 11:09:18,745][TRACE][transport.netty ] [ny01] close connection exception caught on transport layer [[id: 0x1cadc15b, /1.2.3.4:54550 => /1.2.3.4:9500]], disconnecting from relevant node
java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
at sun.nio.ch.IOUtil.read(IOUtil.java:169)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:1.2.3.4:312)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:1.2.3.4:42)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment