Skip to content

Instantly share code, notes, and snippets.

@dbenson
Created February 24, 2011 16:22
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dbenson/842370 to your computer and use it in GitHub Desktop.
Save dbenson/842370 to your computer and use it in GitHub Desktop.
Cluster partition and loss of data
[2011-02-24 02:16:58,618][INFO ][discovery.zen ] [dm-essearchp101.bldrprod.local-ElasticSearch] master_left [[dm-essearchp104.bldrprod.local-ElasticSearch][K7mlyvnETFu2bqY1mjn6dw][inet[/10.15.104.4:9300]]], reason [failed to ping, tried [3] times, each with maximum [30s] timeout]
[2011-02-24 02:16:58,635][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] master {new [dm-essearchp101.bldrprod.local-ElasticSearch][UZ5LMbXdR4i1yY-EHbzxaQ][inet[/10.15.104.1:9300]], previous [dm-essearchp104.bldrprod.local-ElasticSearch][K7mlyvnETFu2bqY1mjn6dw][inet[/10.15.104.4:9300]]}, removed {[dm-essearchp104.bldrprod.local-ElasticSearch][K7mlyvnETFu2bqY1mjn6dw][inet[/10.15.104.4:9300]],}, reason: zen-disco-master_failed ([dm-essearchp104.bldrprod.local-ElasticSearch][K7mlyvnETFu2bqY1mjn6dw][inet[/10.15.104.4:9300]])
[2011-02-24 02:20:20,367][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]]], id [14262659]
[2011-02-24 02:20:20,367][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]], id [14262660]
[2011-02-24 02:20:24,289][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]]], id [14262327]
[2011-02-24 02:20:24,289][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]]], id [14262404]
[2011-02-24 02:20:24,290][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]]], id [14262472]
[2011-02-24 02:20:24,292][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]], id [14262331]
[2011-02-24 02:20:24,292][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]], id [14262399]
[2011-02-24 02:20:24,292][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]], id [14262474]
[2011-02-24 02:20:24,300][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]]], id [14262325]
[2011-02-24 02:20:24,301][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]], id [14262326]
[2011-02-24 02:20:26,336][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp104.bldrprod.local-ESIndexer][fRRH8vRMTnu6SQI79rpHig][inet[/10.15.104.4:9301]]{client=true, data=false}], id [14262333]
[2011-02-24 02:20:26,336][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp104.bldrprod.local-ESIndexer][fRRH8vRMTnu6SQI79rpHig][inet[/10.15.104.4:9301]]{client=true, data=false}], id [14262402]
[2011-02-24 02:20:26,336][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp104.bldrprod.local-ESIndexer][fRRH8vRMTnu6SQI79rpHig][inet[/10.15.104.4:9301]]{client=true, data=false}], id [14262471]
[2011-02-24 02:20:31,451][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp103.bldrprod.local-ESIndexer][M2tK7q1US2WjO6kK7IPNlw][inet[/10.15.104.3:9302]]{client=true, data=false}], id [14262336]
[2011-02-24 02:20:31,451][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp103.bldrprod.local-ESIndexer][M2tK7q1US2WjO6kK7IPNlw][inet[/10.15.104.3:9302]]{client=true, data=false}], id [14262405]
[2011-02-24 02:20:31,451][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp103.bldrprod.local-ESIndexer][M2tK7q1US2WjO6kK7IPNlw][inet[/10.15.104.3:9302]]{client=true, data=false}], id [14262475]
[2011-02-24 02:20:32,477][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp102.bldrprod.local-essearcherserver][6QttpJ6DR227OYRnKCxnhQ][inet[/10.15.104.2:9301]]{client=true, data=false}], id [14262328]
[2011-02-24 02:20:32,477][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp102.bldrprod.local-essearcherserver][6QttpJ6DR227OYRnKCxnhQ][inet[/10.15.104.2:9301]]{client=true, data=false}], id [14262400]
[2011-02-24 02:20:32,477][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp102.bldrprod.local-essearcherserver][6QttpJ6DR227OYRnKCxnhQ][inet[/10.15.104.2:9301]]{client=true, data=false}], id [14262470]
[2011-02-24 02:20:33,499][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp103.bldrprod.local-essearcherserver][-xiqLgsxTm2Dxl-oEQTvUA][inet[/10.15.104.3:9301]]{client=true, data=false}], id [14262332]
[2011-02-24 02:20:33,499][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp103.bldrprod.local-essearcherserver][-xiqLgsxTm2Dxl-oEQTvUA][inet[/10.15.104.3:9301]]{client=true, data=false}], id [14262401]
[2011-02-24 02:20:33,499][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp103.bldrprod.local-essearcherserver][-xiqLgsxTm2Dxl-oEQTvUA][inet[/10.15.104.3:9301]]{client=true, data=false}], id [14262476]
[2011-02-24 02:20:35,544][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp104.bldrprod.local-essearcherserver][4zVgAwBmRlGQHXXXhwfiNw][inet[/10.15.104.4:9302]]{client=true, data=false}], id [14262334]
[2011-02-24 02:20:35,545][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp104.bldrprod.local-essearcherserver][4zVgAwBmRlGQHXXXhwfiNw][inet[/10.15.104.4:9302]]{client=true, data=false}], id [14262406]
[2011-02-24 02:20:35,545][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp104.bldrprod.local-essearcherserver][4zVgAwBmRlGQHXXXhwfiNw][inet[/10.15.104.4:9302]]{client=true, data=false}], id [14262469]
[2011-02-24 02:20:41,723][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]]], id [14262597]
[2011-02-24 02:20:41,723][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]], id [14262598]
[2011-02-24 02:20:54,343][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]], id [14262396]
[2011-02-24 02:20:55,366][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]]], id [14262395]
[2011-02-24 02:20:59,570][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp102.bldrprod.local-ESIndexer][WOSuCf1ASd6ukY_wppPJsw][inet[/10.15.104.2:9302]]{client=true, data=false}], id [14262329]
[2011-02-24 02:20:59,570][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp102.bldrprod.local-ESIndexer][WOSuCf1ASd6ukY_wppPJsw][inet[/10.15.104.2:9302]]{client=true, data=false}], id [14262403]
[2011-02-24 02:20:59,570][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [discovery/zen/fd/ping], node [[dm-essearchp102.bldrprod.local-ESIndexer][WOSuCf1ASd6ukY_wppPJsw][inet[/10.15.104.2:9302]]{client=true, data=false}], id [14262473]
[2011-02-24 02:21:24,440][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]]], id [14262465]
[2011-02-24 02:21:24,440][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]]], id [14262793]
[2011-02-24 02:21:27,508][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]], id [14262466]
[2011-02-24 02:21:27,508][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]], id [14262794]
[2011-02-24 02:21:27,508][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]], id [14262948]
[2011-02-24 02:21:54,539][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]]], id [14262535]
[2011-02-24 02:21:54,540][WARN ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] Received response for a request that has timed out, action [/cluster/nodes/indices/shard/store/node], node [[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]], id [14262536]
[2011-02-24 02:21:54,834][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp103.bldrprod.local-ESIndexer][M2tK7q1US2WjO6kK7IPNlw][inet[/10.15.104.3:9302]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp103.bldrprod.local-ESIndexer][M2tK7q1US2WjO6kK7IPNlw][inet[/10.15.104.3:9302]]{client=true, data=false}), reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2011-02-24 02:21:54,837][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp102.bldrprod.local-ESIndexer][WOSuCf1ASd6ukY_wppPJsw][inet[/10.15.104.2:9302]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp102.bldrprod.local-ESIndexer][WOSuCf1ASd6ukY_wppPJsw][inet[/10.15.104.2:9302]]{client=true, data=false}), reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2011-02-24 02:21:54,839][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]],}, reason: zen-disco-node_failed([dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2011-02-24 02:21:54,841][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp104.bldrprod.local-ESIndexer][fRRH8vRMTnu6SQI79rpHig][inet[/10.15.104.4:9301]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp104.bldrprod.local-ESIndexer][fRRH8vRMTnu6SQI79rpHig][inet[/10.15.104.4:9301]]{client=true, data=false}), reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2011-02-24 02:21:54,843][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]],}, reason: zen-disco-node_failed([dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2011-02-24 02:21:54,853][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp104.bldrprod.local-essearcherserver][4zVgAwBmRlGQHXXXhwfiNw][inet[/10.15.104.4:9302]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp104.bldrprod.local-essearcherserver][4zVgAwBmRlGQHXXXhwfiNw][inet[/10.15.104.4:9302]]{client=true, data=false}), reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2011-02-24 02:21:54,861][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp103.bldrprod.local-essearcherserver][-xiqLgsxTm2Dxl-oEQTvUA][inet[/10.15.104.3:9301]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp103.bldrprod.local-essearcherserver][-xiqLgsxTm2Dxl-oEQTvUA][inet[/10.15.104.3:9301]]{client=true, data=false}), reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2011-02-24 02:21:54,862][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp102.bldrprod.local-essearcherserver][6QttpJ6DR227OYRnKCxnhQ][inet[/10.15.104.2:9301]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp102.bldrprod.local-essearcherserver][6QttpJ6DR227OYRnKCxnhQ][inet[/10.15.104.2:9301]]{client=true, data=false}), reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2011-02-24 02:21:55,412][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]],}, reason: zen-disco-receive(from node[[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]])
[2011-02-24 02:21:55,462][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp104.bldrprod.local-ESIndexer][fRRH8vRMTnu6SQI79rpHig][inet[/10.15.104.4:9301]]{client=true, data=false},}, reason: zen-disco-receive(from node[[dm-essearchp104.bldrprod.local-ESIndexer][fRRH8vRMTnu6SQI79rpHig][inet[/10.15.104.4:9301]]{client=true, data=false}])
[2011-02-24 02:21:55,516][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]],}, reason: zen-disco-receive(from node[[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]]])
[2011-02-24 02:21:55,584][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp103.bldrprod.local-ESIndexer][M2tK7q1US2WjO6kK7IPNlw][inet[/10.15.104.3:9302]]{client=true, data=false},}, reason: zen-disco-receive(from node[[dm-essearchp103.bldrprod.local-ESIndexer][M2tK7q1US2WjO6kK7IPNlw][inet[/10.15.104.3:9302]]{client=true, data=false}])
[2011-02-24 02:21:55,629][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp102.bldrprod.local-essearcherserver][6QttpJ6DR227OYRnKCxnhQ][inet[/10.15.104.2:9301]]{client=true, data=false},}, reason: zen-disco-receive(from node[[dm-essearchp102.bldrprod.local-essearcherserver][6QttpJ6DR227OYRnKCxnhQ][inet[/10.15.104.2:9301]]{client=true, data=false}])
[2011-02-24 02:21:55,652][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp103.bldrprod.local-essearcherserver][-xiqLgsxTm2Dxl-oEQTvUA][inet[/10.15.104.3:9301]]{client=true, data=false},}, reason: zen-disco-receive(from node[[dm-essearchp103.bldrprod.local-essearcherserver][-xiqLgsxTm2Dxl-oEQTvUA][inet[/10.15.104.3:9301]]{client=true, data=false}])
[2011-02-24 02:21:55,662][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp102.bldrprod.local-ESIndexer][WOSuCf1ASd6ukY_wppPJsw][inet[/10.15.104.2:9302]]{client=true, data=false},}, reason: zen-disco-receive(from node[[dm-essearchp102.bldrprod.local-ESIndexer][WOSuCf1ASd6ukY_wppPJsw][inet[/10.15.104.2:9302]]{client=true, data=false}])
[2011-02-24 02:21:55,667][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp104.bldrprod.local-essearcherserver][4zVgAwBmRlGQHXXXhwfiNw][inet[/10.15.104.4:9302]]{client=true, data=false},}, reason: zen-disco-receive(from node[[dm-essearchp104.bldrprod.local-essearcherserver][4zVgAwBmRlGQHXXXhwfiNw][inet[/10.15.104.4:9302]]{client=true, data=false}])
[2011-02-24 02:27:59,487][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp101.bldrprod.local-ESIndexer][Q-mYKiOfTXaeXlxTj1bm_g][inet[/10.15.104.1:9302]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp101.bldrprod.local-ESIndexer][Q-mYKiOfTXaeXlxTj1bm_g][inet[/10.15.104.1:9302]]{client=true, data=false}), reason transport disconnected (with verified connect)
[2011-02-24 03:01:42,738][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp101.bldrprod.local-essearcherserver][7r2eKgQZRNOsV4740Yed2Q][inet[/10.15.104.1:9301]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp101.bldrprod.local-essearcherserver][7r2eKgQZRNOsV4740Yed2Q][inet[/10.15.104.1:9301]]{client=true, data=false}), reason transport disconnected (with verified connect)
[2011-02-24 03:01:56,348][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp102.bldrprod.local-essearcherserver][6QttpJ6DR227OYRnKCxnhQ][inet[/10.15.104.2:9301]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp102.bldrprod.local-essearcherserver][6QttpJ6DR227OYRnKCxnhQ][inet[/10.15.104.2:9301]]{client=true, data=false}), reason transport disconnected (with verified connect)
[2011-02-24 03:02:19,295][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp103.bldrprod.local-essearcherserver][-xiqLgsxTm2Dxl-oEQTvUA][inet[/10.15.104.3:9301]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp103.bldrprod.local-essearcherserver][-xiqLgsxTm2Dxl-oEQTvUA][inet[/10.15.104.3:9301]]{client=true, data=false}), reason transport disconnected (with verified connect)
[2011-02-24 03:02:56,003][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp104.bldrprod.local-essearcherserver][4zVgAwBmRlGQHXXXhwfiNw][inet[/10.15.104.4:9302]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp104.bldrprod.local-essearcherserver][4zVgAwBmRlGQHXXXhwfiNw][inet[/10.15.104.4:9302]]{client=true, data=false}), reason transport disconnected (with verified connect)
[2011-02-24 03:19:27,012][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp103.bldrprod.local-ESIndexer][M2tK7q1US2WjO6kK7IPNlw][inet[/10.15.104.3:9302]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp103.bldrprod.local-ESIndexer][M2tK7q1US2WjO6kK7IPNlw][inet[/10.15.104.3:9302]]{client=true, data=false}), reason transport disconnected (with verified connect)
[2011-02-24 03:20:20,004][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp102.bldrprod.local-ESIndexer][WOSuCf1ASd6ukY_wppPJsw][inet[/10.15.104.2:9302]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp102.bldrprod.local-ESIndexer][WOSuCf1ASd6ukY_wppPJsw][inet[/10.15.104.2:9302]]{client=true, data=false}), reason transport disconnected (with verified connect)
[2011-02-24 03:20:31,361][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] removed {[dm-essearchp104.bldrprod.local-ESIndexer][fRRH8vRMTnu6SQI79rpHig][inet[/10.15.104.4:9301]]{client=true, data=false},}, reason: zen-disco-node_failed([dm-essearchp104.bldrprod.local-ESIndexer][fRRH8vRMTnu6SQI79rpHig][inet[/10.15.104.4:9301]]{client=true, data=false}), reason transport disconnected (with verified connect)
[2011-02-24 03:23:50,000][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[1515]: stopping ...
[2011-02-24 03:23:50,631][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[1515]: stopped
[2011-02-24 03:23:50,631][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[1515]: closing ...
[2011-02-24 03:23:50,853][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[1515]: closed
[2011-02-24 03:35:32,951][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[27292]: initializing ...
[2011-02-24 03:35:32,953][INFO ][plugins ] [dm-essearchp101.bldrprod.local-ElasticSearch] loaded []
[2011-02-24 03:35:34,327][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[27292]: initialized
[2011-02-24 03:35:34,328][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[27292]: starting ...
[2011-02-24 03:35:34,381][INFO ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.15.104.1:9300]}
[2011-02-24 03:35:37,422][WARN ][discovery.zen.ping.multicast] [dm-essearchp101.bldrprod.local-ElasticSearch] received ping response with no matching id [1]
[2011-02-24 03:35:37,422][WARN ][discovery.zen.ping.multicast] [dm-essearchp101.bldrprod.local-ElasticSearch] received ping response with no matching id [1]
[2011-02-24 03:35:37,422][WARN ][discovery.zen.ping.multicast] [dm-essearchp101.bldrprod.local-ElasticSearch] received ping response with no matching id [1]
[2011-02-24 03:35:37,422][WARN ][discovery.zen.ping.multicast] [dm-essearchp101.bldrprod.local-ElasticSearch] received ping response with no matching id [1]
[2011-02-24 03:35:37,423][WARN ][discovery.zen.ping.multicast] [dm-essearchp101.bldrprod.local-ElasticSearch] received ping response with no matching id [1]
[2011-02-24 03:35:37,424][WARN ][discovery.zen.ping.multicast] [dm-essearchp101.bldrprod.local-ElasticSearch] received ping response with no matching id [1]
[2011-02-24 03:35:37,518][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] detected_master [dm-essearchp104.bldrprod.local-ElasticSearch][K7mlyvnETFu2bqY1mjn6dw][inet[/10.15.104.4:9300]], added {[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]],[dm-essearchp104.bldrprod.local-ElasticSearch][K7mlyvnETFu2bqY1mjn6dw][inet[/10.15.104.4:9300]],[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]],}, reason: zen-disco-receive(from [[dm-essearchp104.bldrprod.local-ElasticSearch][K7mlyvnETFu2bqY1mjn6dw][inet[/10.15.104.4:9300]]])
[2011-02-24 03:35:37,524][INFO ][discovery ] [dm-essearchp101.bldrprod.local-ElasticSearch] bldrprod-0.14.0/EYtKduPLSG2jX7owzVUG5Q
[2011-02-24 03:35:37,969][INFO ][http ] [dm-essearchp101.bldrprod.local-ElasticSearch] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.15.104.1:9200]}
[2011-02-24 03:35:37,969][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[27292]: started
[2011-02-24 04:04:15,733][INFO ][action.admin.cluster.node.shutdown] [dm-essearchp101.bldrprod.local-ElasticSearch] shutting down in [200ms]
[2011-02-24 04:04:15,938][INFO ][action.admin.cluster.node.shutdown] [dm-essearchp101.bldrprod.local-ElasticSearch] initiating requested shutdown (using service)
[2011-02-24 04:04:18,074][INFO ][discovery.zen ] [dm-essearchp101.bldrprod.local-ElasticSearch] master_left [[dm-essearchp104.bldrprod.local-ElasticSearch][K7mlyvnETFu2bqY1mjn6dw][inet[/10.15.104.4:9300]]], reason [shut_down]
[2011-02-24 04:04:18,076][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] master {new [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]], previous [dm-essearchp104.bldrprod.local-ElasticSearch][K7mlyvnETFu2bqY1mjn6dw][inet[/10.15.104.4:9300]]}, removed {[dm-essearchp104.bldrprod.local-ElasticSearch][K7mlyvnETFu2bqY1mjn6dw][inet[/10.15.104.4:9300]],}, reason: zen-disco-master_failed ([dm-essearchp104.bldrprod.local-ElasticSearch][K7mlyvnETFu2bqY1mjn6dw][inet[/10.15.104.4:9300]])
[2011-02-24 04:04:18,234][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[27292]: stopping ...
[2011-02-24 04:04:19,023][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [index12_20110110233052][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,023][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [genericnews2_20110110233049][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,025][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [rochdale_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,025][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-ft_20110110233050][2], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,026][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-nab_20110110233050][3], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,026][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports1_20110110233051][5], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,026][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports1_20110110233051][3], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,026][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [djnf_20110110233049][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,026][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports1_20110110233051][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,026][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [index05_20110110233051][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,026][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-maestro-report1_20110110233050][4], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,026][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [rss_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,026][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports1_20110110233051][2], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,027][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [genericnews_20110110233049][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,026][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-ft_20110110233050][4], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,027][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-nab_20110110233050][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,027][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports2_20110110233051][3], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,027][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports1_20110110233051][6], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,027][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [genericreports_20110110233050][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,026][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports2_20110110233051][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,026][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-nonenglish_20110110233050][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,026][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [cbsmw_20110110233048][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,028][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [index23_20110110233054][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,027][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [sec_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,027][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [moodys_20110110233055][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,027][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [qmalerts_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,237][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [ftalerts_20110110233049][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,237][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports1_20110110233051][2], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,237][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports1_20110110233051][3], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,237][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [genericnews_20110110233049][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,237][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [genericreports_20110110233050][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,237][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports1_20110110233051][5], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,237][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-maestro-report1_20110110233050][3], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,237][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [sec_20110110233056][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,237][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [rss_20110110233056][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,237][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports1_20110110233051][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,237][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports1_20110110233051][6], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,237][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [qmalerts_20110110233056][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,238][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [index19_20110110233053][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,237][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-maestro-report1_20110110233050][4], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,238][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [index27_20110110233054][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,238][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [stdpoor_20110110233057][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,238][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [wachovia_20110110233057][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,238][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-nab_20110110233050][2], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,238][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [index24_20110110233054][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,239][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [neddavis_20110110233055][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,239][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [briefing_20110110233047][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,239][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [bia_20110110233046][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,239][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [index22_20110110233053][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,239][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports2_20110110233051][1], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,239][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [idol-reports2_20110110233051][4], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:19,239][WARN ][cluster.action.shard ] [dm-essearchp101.bldrprod.local-ElasticSearch] received shard failed for [index07_20110110233052][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED], reason [master [dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]] marked shard as started, but shard have not been created, mark shard as failed]
[2011-02-24 04:04:28,256][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[27292]: stopped
[2011-02-24 04:04:28,256][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[27292]: closing ...
[2011-02-24 04:04:28,745][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[27292]: closed
[2011-02-24 04:04:28,746][WARN ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] failed to execute cluster state update, state:
version [2189], source [routing-table-updater]
nodes:
[dm-essearchp102.bldrprod.local-ElasticSearch][WEv2yxs-S2ihrtbrdoyyvw][inet[/10.15.104.2:9300]]
[dm-essearchp101.bldrprod.local-ElasticSearch][EYtKduPLSG2jX7owzVUG5Q][inet[/10.15.104.1:9300]], local, master
[dm-essearchp103.bldrprod.local-ElasticSearch][fRPn3o62TPmtCLUHPeOxNg][inet[/10.15.104.3:9300]]
routing_table:
-- index [genericnews2_20110110233049]
----shard_id [genericnews2_20110110233049][0]
--------[genericnews2_20110110233049][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[genericnews2_20110110233049][0], node[K7mlyvnETFu2bqY1mjn6dw], relocating [fRPn3o62TPmtCLUHPeOxNg], [R], s[RELOCATING]
-- index [index16_20110110233053]
----shard_id [index16_20110110233053][0]
--------[index16_20110110233053][0], node[null], [R], s[UNASSIGNED]
--------[index16_20110110233053][0], node[null], [R], s[UNASSIGNED]
-- index [goldman_20110110233050]
----shard_id [goldman_20110110233050][0]
--------[goldman_20110110233050][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[goldman_20110110233050][0], node[EYtKduPLSG2jX7owzVUG5Q], [P], s[STARTED]
-- index [index12_20110110233052]
----shard_id [index12_20110110233052][0]
--------[index12_20110110233052][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[index12_20110110233052][0], node[K7mlyvnETFu2bqY1mjn6dw], relocating [EYtKduPLSG2jX7owzVUG5Q], [R], s[RELOCATING]
-- index [spider01_20110110233057]
----shard_id [spider01_20110110233057][0]
--------[spider01_20110110233057][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[spider01_20110110233057][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [rbcvideo_20110110233056]
----shard_id [rbcvideo_20110110233056][0]
--------[rbcvideo_20110110233056][0], node[null], [R], s[UNASSIGNED]
--------[rbcvideo_20110110233056][0], node[null], [R], s[UNASSIGNED]
-- index [rbcinst_20110110233056]
----shard_id [rbcinst_20110110233056][0]
--------[rbcinst_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[rbcinst_20110110233056][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
-- index [rochdale_20110110233056]
----shard_id [rochdale_20110110233056][0]
--------[rochdale_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[rochdale_20110110233056][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
-- index [ftalerts_20110110233049]
----shard_id [ftalerts_20110110233049][0]
--------[ftalerts_20110110233049][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[ftalerts_20110110233049][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
-- index [index28_20110110233055]
----shard_id [index28_20110110233055][0]
--------[index28_20110110233055][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[index28_20110110233055][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [djcnr_20110110233048]
----shard_id [djcnr_20110110233048][0]
--------[djcnr_20110110233048][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[djcnr_20110110233048][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
-- index [smereports_20110110233056]
----shard_id [smereports_20110110233056][0]
--------[smereports_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[smereports_20110110233056][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
-- index [index21_20110110233053]
----shard_id [index21_20110110233053][0]
--------[index21_20110110233053][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[index21_20110110233053][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
-- index [index18_20110110233053]
----shard_id [index18_20110110233053][0]
--------[index18_20110110233053][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[index18_20110110233053][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
-- index [index29_20110110233055]
----shard_id [index29_20110110233055][0]
--------[index29_20110110233055][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[index29_20110110233055][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [idol-ft_20110110233050]
----shard_id [idol-ft_20110110233050][0]
--------[idol-ft_20110110233050][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-ft_20110110233050][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
----shard_id [idol-ft_20110110233050][1]
--------[idol-ft_20110110233050][1], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[idol-ft_20110110233050][1], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
----shard_id [idol-ft_20110110233050][2]
--------[idol-ft_20110110233050][2], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-ft_20110110233050][2], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
----shard_id [idol-ft_20110110233050][3]
--------[idol-ft_20110110233050][3], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[idol-ft_20110110233050][3], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
----shard_id [idol-ft_20110110233050][4]
--------[idol-ft_20110110233050][4], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-ft_20110110233050][4], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
-- index [genericnews_20110110233049]
----shard_id [genericnews_20110110233049][0]
--------[genericnews_20110110233049][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[genericnews_20110110233049][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [index15_20110110233052]
----shard_id [index15_20110110233052][0]
--------[index15_20110110233052][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[index15_20110110233052][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
-- index [genericreports_20110110233050]
----shard_id [genericreports_20110110233050][0]
--------[genericreports_20110110233050][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[genericreports_20110110233050][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
-- index [qmalerts_20110110233056]
----shard_id [qmalerts_20110110233056][0]
--------[qmalerts_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[qmalerts_20110110233056][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
-- index [idol-reports1_20110110233051]
----shard_id [idol-reports1_20110110233051][0]
--------[idol-reports1_20110110233051][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-reports1_20110110233051][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
----shard_id [idol-reports1_20110110233051][1]
--------[idol-reports1_20110110233051][1], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-reports1_20110110233051][1], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
----shard_id [idol-reports1_20110110233051][2]
--------[idol-reports1_20110110233051][2], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-reports1_20110110233051][2], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
----shard_id [idol-reports1_20110110233051][3]
--------[idol-reports1_20110110233051][3], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-reports1_20110110233051][3], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
----shard_id [idol-reports1_20110110233051][4]
--------[idol-reports1_20110110233051][4], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[idol-reports1_20110110233051][4], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
----shard_id [idol-reports1_20110110233051][5]
--------[idol-reports1_20110110233051][5], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-reports1_20110110233051][5], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
----shard_id [idol-reports1_20110110233051][6]
--------[idol-reports1_20110110233051][6], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-reports1_20110110233051][6], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
----shard_id [idol-reports1_20110110233051][7]
--------[idol-reports1_20110110233051][7], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-reports1_20110110233051][7], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
-- index [rss_20110110233056]
----shard_id [rss_20110110233056][0]
--------[rss_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[rss_20110110233056][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [sec_20110110233056]
----shard_id [sec_20110110233056][0]
--------[sec_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[sec_20110110233056][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
-- index [index25_20110110233054]
----shard_id [index25_20110110233054][0]
--------[index25_20110110233054][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[index25_20110110233054][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
-- index [comtex_20110110233048]
----shard_id [comtex_20110110233048][0]
--------[comtex_20110110233048][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[comtex_20110110233048][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [idol-maestro-report1_20110110233050]
----shard_id [idol-maestro-report1_20110110233050][0]
--------[idol-maestro-report1_20110110233050][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[idol-maestro-report1_20110110233050][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
----shard_id [idol-maestro-report1_20110110233050][1]
--------[idol-maestro-report1_20110110233050][1], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-maestro-report1_20110110233050][1], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
----shard_id [idol-maestro-report1_20110110233050][2]
--------[idol-maestro-report1_20110110233050][2], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-maestro-report1_20110110233050][2], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
----shard_id [idol-maestro-report1_20110110233050][3]
--------[idol-maestro-report1_20110110233050][3], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-maestro-report1_20110110233050][3], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
----shard_id [idol-maestro-report1_20110110233050][4]
--------[idol-maestro-report1_20110110233050][4], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-maestro-report1_20110110233050][4], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
-- index [index13_20110110233052]
----shard_id [index13_20110110233052][0]
--------[index13_20110110233052][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[index13_20110110233052][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [index19_20110110233053]
----shard_id [index19_20110110233053][0]
--------[index19_20110110233053][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index19_20110110233053][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [index01_20110110233051]
----shard_id [index01_20110110233051][0]
--------[index01_20110110233051][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index01_20110110233051][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [wachovia_20110110233057]
----shard_id [wachovia_20110110233057][0]
--------[wachovia_20110110233057][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[wachovia_20110110233057][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [index08_20110110233052]
----shard_id [index08_20110110233052][0]
--------[index08_20110110233052][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[index08_20110110233052][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [newsmedia_20110110233055]
----shard_id [newsmedia_20110110233055][0]
--------[newsmedia_20110110233055][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[newsmedia_20110110233055][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [index27_20110110233054]
----shard_id [index27_20110110233054][0]
--------[index27_20110110233054][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index27_20110110233054][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [cbsmw_20110110233048]
----shard_id [cbsmw_20110110233048][0]
--------[cbsmw_20110110233048][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[cbsmw_20110110233048][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [scottrade_20110110233056]
----shard_id [scottrade_20110110233056][0]
--------[scottrade_20110110233056][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[scottrade_20110110233056][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [stdpoor_20110110233057]
----shard_id [stdpoor_20110110233057][0]
--------[stdpoor_20110110233057][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[stdpoor_20110110233057][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [idol-nab_20110110233050]
----shard_id [idol-nab_20110110233050][0]
--------[idol-nab_20110110233050][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-nab_20110110233050][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
----shard_id [idol-nab_20110110233050][1]
--------[idol-nab_20110110233050][1], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-nab_20110110233050][1], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
----shard_id [idol-nab_20110110233050][2]
--------[idol-nab_20110110233050][2], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-nab_20110110233050][2], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
----shard_id [idol-nab_20110110233050][3]
--------[idol-nab_20110110233050][3], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-nab_20110110233050][3], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
----shard_id [idol-nab_20110110233050][4]
--------[idol-nab_20110110233050][4], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-nab_20110110233050][4], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [index24_20110110233054]
----shard_id [index24_20110110233054][0]
--------[index24_20110110233054][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index24_20110110233054][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [index23_20110110233054]
----shard_id [index23_20110110233054][0]
--------[index23_20110110233054][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[index23_20110110233054][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [wachovia_archive_20110110233057]
----shard_id [wachovia_archive_20110110233057][0]
--------[wachovia_archive_20110110233057][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[wachovia_archive_20110110233057][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [neddavis_20110110233055]
----shard_id [neddavis_20110110233055][0]
--------[neddavis_20110110233055][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[neddavis_20110110233055][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [dorsey_20110110233049]
----shard_id [dorsey_20110110233049][0]
--------[dorsey_20110110233049][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[dorsey_20110110233049][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [idol-nonenglish_20110110233050]
----shard_id [idol-nonenglish_20110110233050][0]
--------[idol-nonenglish_20110110233050][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-nonenglish_20110110233050][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [briefing_20110110233047]
----shard_id [briefing_20110110233047][0]
--------[briefing_20110110233047][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[briefing_20110110233047][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [bloomberg_20110110233047]
----shard_id [bloomberg_20110110233047][0]
--------[bloomberg_20110110233047][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[bloomberg_20110110233047][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [moodys_20110110233055]
----shard_id [moodys_20110110233055][0]
--------[moodys_20110110233055][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[moodys_20110110233055][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [csfb1_20110110233048]
----shard_id [csfb1_20110110233048][0]
--------[csfb1_20110110233048][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[csfb1_20110110233048][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [bia_20110110233046]
----shard_id [bia_20110110233046][0]
--------[bia_20110110233046][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[bia_20110110233046][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [index22_20110110233053]
----shard_id [index22_20110110233053][0]
--------[index22_20110110233053][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index22_20110110233053][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [index05_20110110233051]
----shard_id [index05_20110110233051][0]
--------[index05_20110110233051][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[index05_20110110233051][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [lehman_20110110233055]
----shard_id [lehman_20110110233055][0]
--------[lehman_20110110233055][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[lehman_20110110233055][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [idol-reports2_20110110233051]
----shard_id [idol-reports2_20110110233051][0]
--------[idol-reports2_20110110233051][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-reports2_20110110233051][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
----shard_id [idol-reports2_20110110233051][1]
--------[idol-reports2_20110110233051][1], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-reports2_20110110233051][1], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
----shard_id [idol-reports2_20110110233051][2]
--------[idol-reports2_20110110233051][2], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-reports2_20110110233051][2], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
----shard_id [idol-reports2_20110110233051][3]
--------[idol-reports2_20110110233051][3], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-reports2_20110110233051][3], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
----shard_id [idol-reports2_20110110233051][4]
--------[idol-reports2_20110110233051][4], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-reports2_20110110233051][4], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [index10_20110110233052]
----shard_id [index10_20110110233052][0]
--------[index10_20110110233052][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index10_20110110233052][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
-- index [djnf_20110110233049]
----shard_id [djnf_20110110233049][0]
--------[djnf_20110110233049][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[djnf_20110110233049][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-- index [index07_20110110233052]
----shard_id [index07_20110110233052][0]
--------[index07_20110110233052][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index07_20110110233052][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
routing_nodes:
-----node_id[WEv2yxs-S2ihrtbrdoyyvw]
--------[genericnews2_20110110233049][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[goldman_20110110233050][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[index12_20110110233052][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[spider01_20110110233057][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[rbcinst_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[rochdale_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[index28_20110110233055][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[smereports_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[index21_20110110233053][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[index29_20110110233055][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-ft_20110110233050][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-ft_20110110233050][2], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-ft_20110110233050][4], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[genericnews_20110110233049][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[genericreports_20110110233050][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[qmalerts_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-reports1_20110110233051][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-reports1_20110110233051][1], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-reports1_20110110233051][2], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-reports1_20110110233051][3], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-reports1_20110110233051][5], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-reports1_20110110233051][6], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-reports1_20110110233051][7], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[rss_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[sec_20110110233056][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[comtex_20110110233048][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-maestro-report1_20110110233050][1], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-maestro-report1_20110110233050][2], node[WEv2yxs-S2ihrtbrdoyyvw], [P], s[STARTED]
--------[idol-maestro-report1_20110110233050][4], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[index13_20110110233052][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[index08_20110110233052][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[newsmedia_20110110233055][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[cbsmw_20110110233048][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-nab_20110110233050][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-nab_20110110233050][1], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-nab_20110110233050][3], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[index23_20110110233054][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[wachovia_archive_20110110233057][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[dorsey_20110110233049][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-nonenglish_20110110233050][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[moodys_20110110233055][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[csfb1_20110110233048][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[index05_20110110233051][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-reports2_20110110233051][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-reports2_20110110233051][2], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[idol-reports2_20110110233051][3], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
--------[djnf_20110110233049][0], node[WEv2yxs-S2ihrtbrdoyyvw], [R], s[STARTED]
-----node_id[EYtKduPLSG2jX7owzVUG5Q]
--------[goldman_20110110233050][0], node[EYtKduPLSG2jX7owzVUG5Q], [P], s[STARTED]
--------[index12_20110110233052][0], node[EYtKduPLSG2jX7owzVUG5Q], relocating [K7mlyvnETFu2bqY1mjn6dw], [R], s[INITIALIZING]
-----node_id[K7mlyvnETFu2bqY1mjn6dw]
--------[genericnews2_20110110233049][0], node[K7mlyvnETFu2bqY1mjn6dw], relocating [fRPn3o62TPmtCLUHPeOxNg], [R], s[RELOCATING]
--------[index12_20110110233052][0], node[K7mlyvnETFu2bqY1mjn6dw], relocating [EYtKduPLSG2jX7owzVUG5Q], [R], s[RELOCATING]
--------[spider01_20110110233057][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[rochdale_20110110233056][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[ftalerts_20110110233049][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[djcnr_20110110233048][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[smereports_20110110233056][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[index18_20110110233053][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[index29_20110110233055][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-ft_20110110233050][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[idol-ft_20110110233050][1], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[idol-ft_20110110233050][3], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[index15_20110110233052][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[idol-reports1_20110110233051][1], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[idol-reports1_20110110233051][4], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[idol-reports1_20110110233051][7], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[index25_20110110233054][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[comtex_20110110233048][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-maestro-report1_20110110233050][0], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[idol-maestro-report1_20110110233050][1], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[idol-maestro-report1_20110110233050][2], node[K7mlyvnETFu2bqY1mjn6dw], [R], s[STARTED]
--------[idol-maestro-report1_20110110233050][3], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index13_20110110233052][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index19_20110110233053][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index01_20110110233051][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[wachovia_20110110233057][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index08_20110110233052][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[newsmedia_20110110233055][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index27_20110110233054][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[cbsmw_20110110233048][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[scottrade_20110110233056][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[stdpoor_20110110233057][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-nab_20110110233050][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-nab_20110110233050][1], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-nab_20110110233050][2], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-nab_20110110233050][3], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-nab_20110110233050][4], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index24_20110110233054][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index23_20110110233054][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[wachovia_archive_20110110233057][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[neddavis_20110110233055][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[dorsey_20110110233049][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-nonenglish_20110110233050][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[briefing_20110110233047][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[bloomberg_20110110233047][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[moodys_20110110233055][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[csfb1_20110110233048][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[bia_20110110233046][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index22_20110110233053][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index05_20110110233051][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[lehman_20110110233055][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-reports2_20110110233051][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-reports2_20110110233051][1], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-reports2_20110110233051][2], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-reports2_20110110233051][3], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[idol-reports2_20110110233051][4], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index10_20110110233052][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[djnf_20110110233049][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
--------[index07_20110110233052][0], node[K7mlyvnETFu2bqY1mjn6dw], [P], s[STARTED]
-----node_id[fRPn3o62TPmtCLUHPeOxNg]
--------[genericnews2_20110110233049][0], node[fRPn3o62TPmtCLUHPeOxNg], relocating [K7mlyvnETFu2bqY1mjn6dw], [R], s[INITIALIZING]
--------[rbcinst_20110110233056][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[ftalerts_20110110233049][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[index28_20110110233055][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[djcnr_20110110233048][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[index21_20110110233053][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[index18_20110110233053][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[idol-ft_20110110233050][1], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[idol-ft_20110110233050][2], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[idol-ft_20110110233050][3], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[idol-ft_20110110233050][4], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[genericnews_20110110233049][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[index15_20110110233052][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[genericreports_20110110233050][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[qmalerts_20110110233056][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[idol-reports1_20110110233051][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[idol-reports1_20110110233051][2], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[idol-reports1_20110110233051][3], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[idol-reports1_20110110233051][4], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[idol-reports1_20110110233051][5], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[idol-reports1_20110110233051][6], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[rss_20110110233056][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[sec_20110110233056][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[index25_20110110233054][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[idol-maestro-report1_20110110233050][0], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[idol-maestro-report1_20110110233050][3], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[idol-maestro-report1_20110110233050][4], node[fRPn3o62TPmtCLUHPeOxNg], [P], s[STARTED]
--------[index19_20110110233053][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[index01_20110110233051][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[wachovia_20110110233057][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[index27_20110110233054][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[scottrade_20110110233056][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[stdpoor_20110110233057][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[idol-nab_20110110233050][2], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[idol-nab_20110110233050][4], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[index24_20110110233054][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[neddavis_20110110233055][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[briefing_20110110233047][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[bloomberg_20110110233047][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[bia_20110110233046][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[index22_20110110233053][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[lehman_20110110233055][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[idol-reports2_20110110233051][1], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[idol-reports2_20110110233051][4], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[index10_20110110233052][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
--------[index07_20110110233052][0], node[fRPn3o62TPmtCLUHPeOxNg], [R], s[STARTED]
---- unassigned
--------[index16_20110110233053][0], node[null], [R], s[UNASSIGNED]
--------[index16_20110110233053][0], node[null], [R], s[UNASSIGNED]
--------[rbcvideo_20110110233056][0], node[null], [R], s[UNASSIGNED]
--------[rbcvideo_20110110233056][0], node[null], [R], s[UNASSIGNED]
org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:49)
at org.elasticsearch.gateway.local.LocalGatewayNodeAllocation.buildShardStores(LocalGatewayNodeAllocation.java:349)
at org.elasticsearch.gateway.local.LocalGatewayNodeAllocation.allocateUnassigned(LocalGatewayNodeAllocation.java:263)
at org.elasticsearch.cluster.routing.allocation.NodeAllocations.allocateUnassigned(NodeAllocations.java:78)
at org.elasticsearch.cluster.routing.allocation.ShardsAllocation.reroute(ShardsAllocation.java:122)
at org.elasticsearch.cluster.routing.allocation.ShardsAllocation.reroute(ShardsAllocation.java:100)
at org.elasticsearch.cluster.routing.RoutingService$RoutingTableUpdater$1.execute(RoutingService.java:124)
at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:171)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at org.elasticsearch.common.util.concurrent.AbstractFuture$Sync.complete(AbstractFuture.java:320)
at org.elasticsearch.common.util.concurrent.AbstractFuture$Sync.setException(AbstractFuture.java:297)
at org.elasticsearch.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:129)
at org.elasticsearch.action.support.AdapterActionFuture.onFailure(AdapterActionFuture.java:87)
at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:56)
at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:43)
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:76)
... 10 more
Caused by: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.elasticsearch.threadpool.support.AbstractThreadPool.execute(AbstractThreadPool.java:124)
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:198)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.start(TransportNodesOperationAction.java:158)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$300(TransportNodesOperationAction.java:92)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction.doExecute(TransportNodesOperationAction.java:67)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction.doExecute(TransportNodesOperationAction.java:44)
at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:54)
... 12 more
[2011-02-24 04:06:03,003][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[28199]: initializing ...
[2011-02-24 04:06:03,005][INFO ][plugins ] [dm-essearchp101.bldrprod.local-ElasticSearch] loaded []
[2011-02-24 04:06:04,141][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[28199]: initialized
[2011-02-24 04:06:04,141][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[28199]: starting ...
[2011-02-24 04:06:04,189][INFO ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.15.104.1:9300]}
[2011-02-24 04:06:16,260][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] new_master [dm-essearchp101.bldrprod.local-ElasticSearch][7BAHaS0MR8epLOSF5z-RYg][inet[/10.15.104.1:9300]], reason: zen-disco-join (elected_as_master)
[2011-02-24 04:06:16,262][INFO ][discovery ] [dm-essearchp101.bldrprod.local-ElasticSearch] bldrprod-0.14.0/7BAHaS0MR8epLOSF5z-RYg
[2011-02-24 04:06:16,475][INFO ][http ] [dm-essearchp101.bldrprod.local-ElasticSearch] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.15.104.1:9200]}
[2011-02-24 04:06:16,476][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[28199]: started
[2011-02-24 04:06:20,672][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp102.bldrprod.local-ElasticSearch][PZZlhTxSTk6UQ3u0jXnuiA][inet[/10.15.104.2:9300]],}, reason: zen-disco-receive(from node[[dm-essearchp102.bldrprod.local-ElasticSearch][PZZlhTxSTk6UQ3u0jXnuiA][inet[/10.15.104.2:9300]]])
[2011-02-24 04:06:25,442][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp103.bldrprod.local-ElasticSearch][za2eUCAKRdO9TC9E-u93og][inet[/10.15.104.3:9300]],}, reason: zen-disco-receive(from node[[dm-essearchp103.bldrprod.local-ElasticSearch][za2eUCAKRdO9TC9E-u93og][inet[/10.15.104.3:9300]]])
[2011-02-24 04:06:31,535][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp104.bldrprod.local-ElasticSearch][sEj7PpgHRx6RwVrGKU3ZxA][inet[/10.15.104.4:9300]],}, reason: zen-disco-receive(from node[[dm-essearchp104.bldrprod.local-ElasticSearch][sEj7PpgHRx6RwVrGKU3ZxA][inet[/10.15.104.4:9300]]])
[2011-02-24 04:06:32,351][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [genericnews2_20110110233049] creating index, cause [gateway], shards [1]/[1], mappings [genericnews2]
[2011-02-24 04:06:32,569][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index16_20110110233053] creating index, cause [gateway], shards [1]/[1], mappings [index16]
[2011-02-24 04:06:32,762][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index12_20110110233052] creating index, cause [gateway], shards [1]/[1], mappings [index12]
[2011-02-24 04:06:32,948][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [goldman_20110110233050] creating index, cause [gateway], shards [1]/[1], mappings [goldman]
[2011-02-24 04:06:33,102][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [spider01_20110110233057] creating index, cause [gateway], shards [1]/[1], mappings [spider01]
[2011-02-24 04:06:33,248][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [rbcvideo_20110110233056] creating index, cause [gateway], shards [1]/[1], mappings [rbcvideo]
[2011-02-24 04:06:33,388][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [rbcinst_20110110233056] creating index, cause [gateway], shards [1]/[1], mappings [rbcinst]
[2011-02-24 04:06:33,516][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [rochdale_20110110233056] creating index, cause [gateway], shards [1]/[1], mappings [rochdale]
[2011-02-24 04:06:33,632][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [ftalerts_20110110233049] creating index, cause [gateway], shards [1]/[1], mappings [ftalerts]
[2011-02-24 04:06:33,744][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index28_20110110233055] creating index, cause [gateway], shards [1]/[1], mappings [index28]
[2011-02-24 04:06:33,849][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [djcnr_20110110233048] creating index, cause [gateway], shards [1]/[1], mappings [djcnr]
[2011-02-24 04:06:33,962][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [smereports_20110110233056] creating index, cause [gateway], shards [1]/[1], mappings [smereports]
[2011-02-24 04:06:34,054][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index21_20110110233053] creating index, cause [gateway], shards [1]/[1], mappings [index21]
[2011-02-24 04:06:34,144][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index18_20110110233053] creating index, cause [gateway], shards [1]/[1], mappings [index18]
[2011-02-24 04:06:34,233][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index29_20110110233055] creating index, cause [gateway], shards [1]/[1], mappings [index29]
[2011-02-24 04:06:34,330][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [idol-ft_20110110233050] creating index, cause [gateway], shards [5]/[1], mappings [idol-ft]
[2011-02-24 04:06:34,417][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [genericnews_20110110233049] creating index, cause [gateway], shards [1]/[1], mappings [genericnews]
[2011-02-24 04:06:34,501][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index15_20110110233052] creating index, cause [gateway], shards [1]/[1], mappings [index15]
[2011-02-24 04:06:34,586][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [genericreports_20110110233050] creating index, cause [gateway], shards [1]/[1], mappings [genericreports]
[2011-02-24 04:06:34,667][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [qmalerts_20110110233056] creating index, cause [gateway], shards [1]/[1], mappings [qmalerts]
[2011-02-24 04:06:34,749][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [idol-reports1_20110110233051] creating index, cause [gateway], shards [8]/[1], mappings [idol-reports1]
[2011-02-24 04:06:34,833][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [sec_20110110233056] creating index, cause [gateway], shards [1]/[1], mappings [sec]
[2011-02-24 04:06:34,913][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [rss_20110110233056] creating index, cause [gateway], shards [1]/[1], mappings [rss]
[2011-02-24 04:06:34,996][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index25_20110110233054] creating index, cause [gateway], shards [1]/[1], mappings [index25]
[2011-02-24 04:06:35,077][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [idol-maestro-report1_20110110233050] creating index, cause [gateway], shards [5]/[1], mappings [idol-maestro-report1]
[2011-02-24 04:06:35,155][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [comtex_20110110233048] creating index, cause [gateway], shards [1]/[1], mappings [comtex]
[2011-02-24 04:06:35,231][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index19_20110110233053] creating index, cause [gateway], shards [1]/[1], mappings [index19]
[2011-02-24 04:06:35,305][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index13_20110110233052] creating index, cause [gateway], shards [1]/[1], mappings [index13]
[2011-02-24 04:06:35,376][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index01_20110110233051] creating index, cause [gateway], shards [1]/[1], mappings [index01]
[2011-02-24 04:06:35,448][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [wachovia_20110110233057] creating index, cause [gateway], shards [1]/[1], mappings [wachovia]
[2011-02-24 04:06:35,522][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index08_20110110233052] creating index, cause [gateway], shards [1]/[1], mappings [index08]
[2011-02-24 04:06:35,593][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [newsmedia_20110110233055] creating index, cause [gateway], shards [1]/[1], mappings [newsmedia]
[2011-02-24 04:06:35,662][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index27_20110110233054] creating index, cause [gateway], shards [1]/[1], mappings [index27]
[2011-02-24 04:06:35,731][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [cbsmw_20110110233048] creating index, cause [gateway], shards [1]/[1], mappings [cbsmw]
[2011-02-24 04:06:35,799][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [scottrade_20110110233056] creating index, cause [gateway], shards [1]/[1], mappings [scottrade]
[2011-02-24 04:06:35,868][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [stdpoor_20110110233057] creating index, cause [gateway], shards [1]/[1], mappings [stdpoor]
[2011-02-24 04:06:35,940][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index24_20110110233054] creating index, cause [gateway], shards [1]/[1], mappings [index24]
[2011-02-24 04:06:36,010][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [idol-nab_20110110233050] creating index, cause [gateway], shards [5]/[1], mappings [idol-nab]
[2011-02-24 04:06:36,079][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index23_20110110233054] creating index, cause [gateway], shards [1]/[1], mappings [index23]
[2011-02-24 04:06:36,147][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [wachovia_archive_20110110233057] creating index, cause [gateway], shards [1]/[1], mappings [wachovia_archive]
[2011-02-24 04:06:36,219][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [neddavis_20110110233055] creating index, cause [gateway], shards [1]/[1], mappings [neddavis]
[2011-02-24 04:06:36,288][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [dorsey_20110110233049] creating index, cause [gateway], shards [1]/[1], mappings [dorsey]
[2011-02-24 04:06:36,356][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [idol-nonenglish_20110110233050] creating index, cause [gateway], shards [1]/[1], mappings [idol-nonenglish]
[2011-02-24 04:06:36,424][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [briefing_20110110233047] creating index, cause [gateway], shards [1]/[1], mappings [briefing]
[2011-02-24 04:06:36,496][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [bloomberg_20110110233047] creating index, cause [gateway], shards [1]/[1], mappings [bloomberg]
[2011-02-24 04:06:36,564][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [moodys_20110110233055] creating index, cause [gateway], shards [1]/[1], mappings [moodys]
[2011-02-24 04:06:36,636][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [bia_20110110233046] creating index, cause [gateway], shards [1]/[1], mappings [bia]
[2011-02-24 04:06:36,705][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [csfb1_20110110233048] creating index, cause [gateway], shards [1]/[1], mappings [csfb1]
[2011-02-24 04:06:36,774][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index22_20110110233053] creating index, cause [gateway], shards [1]/[1], mappings [index22]
[2011-02-24 04:06:36,843][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index05_20110110233051] creating index, cause [gateway], shards [1]/[1], mappings [index05]
[2011-02-24 04:06:36,915][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [lehman_20110110233055] creating index, cause [gateway], shards [1]/[1], mappings [lehman]
[2011-02-24 04:06:36,984][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [idol-reports2_20110110233051] creating index, cause [gateway], shards [5]/[1], mappings [idol-reports2]
[2011-02-24 04:06:37,052][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index10_20110110233052] creating index, cause [gateway], shards [1]/[1], mappings [index10]
[2011-02-24 04:06:37,119][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [djnf_20110110233049] creating index, cause [gateway], shards [1]/[1], mappings [djnf]
[2011-02-24 04:06:37,185][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index07_20110110233052] creating index, cause [gateway], shards [1]/[1], mappings [index07]
[2011-02-24 04:06:37,196][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [genericnews2_20110110233049] created and added to cluster_state
[2011-02-24 04:06:37,201][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index16_20110110233053] created and added to cluster_state
[2011-02-24 04:06:37,206][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index12_20110110233052] created and added to cluster_state
[2011-02-24 04:06:37,211][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [goldman_20110110233050] created and added to cluster_state
[2011-02-24 04:06:37,215][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [spider01_20110110233057] created and added to cluster_state
[2011-02-24 04:06:37,218][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [rbcvideo_20110110233056] created and added to cluster_state
[2011-02-24 04:06:37,222][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [rbcinst_20110110233056] created and added to cluster_state
[2011-02-24 04:06:37,226][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [rochdale_20110110233056] created and added to cluster_state
[2011-02-24 04:06:37,229][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [ftalerts_20110110233049] created and added to cluster_state
[2011-02-24 04:06:37,233][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index28_20110110233055] created and added to cluster_state
[2011-02-24 04:06:37,236][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [djcnr_20110110233048] created and added to cluster_state
[2011-02-24 04:06:37,364][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [smereports_20110110233056] created and added to cluster_state
[2011-02-24 04:06:37,369][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index21_20110110233053] created and added to cluster_state
[2011-02-24 04:06:37,397][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index18_20110110233053] created and added to cluster_state
[2011-02-24 04:06:37,401][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index29_20110110233055] created and added to cluster_state
[2011-02-24 04:06:37,405][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [idol-ft_20110110233050] created and added to cluster_state
[2011-02-24 04:06:37,411][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [genericnews_20110110233049] created and added to cluster_state
[2011-02-24 04:06:37,415][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index15_20110110233052] created and added to cluster_state
[2011-02-24 04:06:37,421][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [genericreports_20110110233050] created and added to cluster_state
[2011-02-24 04:06:37,425][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [qmalerts_20110110233056] created and added to cluster_state
[2011-02-24 04:06:37,429][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [idol-reports1_20110110233051] created and added to cluster_state
[2011-02-24 04:06:37,433][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [sec_20110110233056] created and added to cluster_state
[2011-02-24 04:06:37,437][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [rss_20110110233056] created and added to cluster_state
[2011-02-24 04:06:37,442][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index25_20110110233054] created and added to cluster_state
[2011-02-24 04:06:37,446][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [idol-maestro-report1_20110110233050] created and added to cluster_state
[2011-02-24 04:06:37,451][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [comtex_20110110233048] created and added to cluster_state
[2011-02-24 04:06:37,457][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index19_20110110233053] created and added to cluster_state
[2011-02-24 04:06:37,461][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index13_20110110233052] created and added to cluster_state
[2011-02-24 04:06:37,465][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index01_20110110233051] created and added to cluster_state
[2011-02-24 04:06:37,469][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [wachovia_20110110233057] created and added to cluster_state
[2011-02-24 04:06:37,473][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index08_20110110233052] created and added to cluster_state
[2011-02-24 04:06:37,476][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [newsmedia_20110110233055] created and added to cluster_state
[2011-02-24 04:06:37,480][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index27_20110110233054] created and added to cluster_state
[2011-02-24 04:06:37,484][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [cbsmw_20110110233048] created and added to cluster_state
[2011-02-24 04:06:37,488][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [scottrade_20110110233056] created and added to cluster_state
[2011-02-24 04:06:37,492][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [stdpoor_20110110233057] created and added to cluster_state
[2011-02-24 04:06:37,496][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index24_20110110233054] created and added to cluster_state
[2011-02-24 04:06:37,500][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [idol-nab_20110110233050] created and added to cluster_state
[2011-02-24 04:06:37,504][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index23_20110110233054] created and added to cluster_state
[2011-02-24 04:06:37,508][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [wachovia_archive_20110110233057] created and added to cluster_state
[2011-02-24 04:06:37,512][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [neddavis_20110110233055] created and added to cluster_state
[2011-02-24 04:06:37,518][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [dorsey_20110110233049] created and added to cluster_state
[2011-02-24 04:06:37,522][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [idol-nonenglish_20110110233050] created and added to cluster_state
[2011-02-24 04:06:37,526][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [briefing_20110110233047] created and added to cluster_state
[2011-02-24 04:06:37,531][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [bloomberg_20110110233047] created and added to cluster_state
[2011-02-24 04:06:37,535][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [moodys_20110110233055] created and added to cluster_state
[2011-02-24 04:06:37,539][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [bia_20110110233046] created and added to cluster_state
[2011-02-24 04:06:37,543][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [csfb1_20110110233048] created and added to cluster_state
[2011-02-24 04:06:37,548][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index22_20110110233053] created and added to cluster_state
[2011-02-24 04:06:37,552][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index05_20110110233051] created and added to cluster_state
[2011-02-24 04:06:37,556][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [lehman_20110110233055] created and added to cluster_state
[2011-02-24 04:06:37,561][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [idol-reports2_20110110233051] created and added to cluster_state
[2011-02-24 04:06:37,565][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index10_20110110233052] created and added to cluster_state
[2011-02-24 04:06:37,569][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [djnf_20110110233049] created and added to cluster_state
[2011-02-24 04:06:37,573][INFO ][cluster.metadata ] [dm-essearchp101.bldrprod.local-ElasticSearch] [index07_20110110233052] created and added to cluster_state
[2011-02-24 04:13:50,834][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp101.bldrprod.local-essearcherserver][tCDarOWjQqWR2Y_iPcrkMA][inet[/10.15.104.1:9301]]{client=true, data=false},}, reason: zen-disco-receive(from node[[dm-essearchp101.bldrprod.local-essearcherserver][tCDarOWjQqWR2Y_iPcrkMA][inet[/10.15.104.1:9301]]{client=true, data=false}])
[2011-02-24 04:14:00,904][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp102.bldrprod.local-essearcherserver][OEkAEjifS8e74S38JXYcwg][inet[/10.15.104.2:9301]]{client=true, data=false},}, reason: zen-disco-receive(from node[[dm-essearchp102.bldrprod.local-essearcherserver][OEkAEjifS8e74S38JXYcwg][inet[/10.15.104.2:9301]]{client=true, data=false}])
[2011-02-24 04:14:17,785][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp103.bldrprod.local-essearcherserver][2KQd2bQwSm-oYIpJKkytlQ][inet[/10.15.104.3:9301]]{client=true, data=false},}, reason: zen-disco-receive(from node[[dm-essearchp103.bldrprod.local-essearcherserver][2KQd2bQwSm-oYIpJKkytlQ][inet[/10.15.104.3:9301]]{client=true, data=false}])
[2011-02-24 04:14:57,855][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp104.bldrprod.local-essearcherserver][3LZu42thT-CkNoH29H_mrQ][inet[/10.15.104.4:9301]]{client=true, data=false},}, reason: zen-disco-receive(from node[[dm-essearchp104.bldrprod.local-essearcherserver][3LZu42thT-CkNoH29H_mrQ][inet[/10.15.104.4:9301]]{client=true, data=false}])
[2011-02-24 04:44:56,895][INFO ][action.admin.cluster.node.shutdown] [dm-essearchp101.bldrprod.local-ElasticSearch] [cluster_shutdown]: requested, shutting down in [1s]
[2011-02-24 04:44:58,084][INFO ][action.admin.cluster.node.shutdown] [dm-essearchp101.bldrprod.local-ElasticSearch] [cluster_shutdown]: done shutting down all nodes except master, proceeding to master
[2011-02-24 04:44:58,088][INFO ][action.admin.cluster.node.shutdown] [dm-essearchp101.bldrprod.local-ElasticSearch] shutting down in [200ms]
[2011-02-24 04:44:58,291][INFO ][action.admin.cluster.node.shutdown] [dm-essearchp101.bldrprod.local-ElasticSearch] initiating requested shutdown (using service)
[2011-02-24 04:45:00,706][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[28199]: stopping ...
[2011-02-24 04:45:01,104][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[28199]: stopped
[2011-02-24 04:45:01,105][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[28199]: closing ...
[2011-02-24 04:45:01,601][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[28199]: closed
[2011-02-24 04:46:26,857][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[29623]: initializing ...
[2011-02-24 04:46:26,859][INFO ][plugins ] [dm-essearchp101.bldrprod.local-ElasticSearch] loaded []
[2011-02-24 04:46:28,131][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[29623]: initialized
[2011-02-24 04:46:28,131][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[29623]: starting ...
[2011-02-24 04:46:28,195][INFO ][transport ] [dm-essearchp101.bldrprod.local-ElasticSearch] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.15.104.1:9300]}
[2011-02-24 04:46:34,293][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] detected_master [dm-essearchp104.bldrprod.local-ElasticSearch][Dd3P4wdxR5mvvkZ4D70ZMg][inet[/10.15.104.4:9300]], added {[dm-essearchp104.bldrprod.local-ElasticSearch][Dd3P4wdxR5mvvkZ4D70ZMg][inet[/10.15.104.4:9300]],}, reason: zen-disco-receive(from [[dm-essearchp104.bldrprod.local-ElasticSearch][Dd3P4wdxR5mvvkZ4D70ZMg][inet[/10.15.104.4:9300]]])
[2011-02-24 04:46:34,294][INFO ][discovery ] [dm-essearchp101.bldrprod.local-ElasticSearch] bldrprod-0.14.0/ZWws3rcNSc-mpLBa-R0AiQ
[2011-02-24 04:46:34,725][INFO ][http ] [dm-essearchp101.bldrprod.local-ElasticSearch] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.15.104.1:9200]}
[2011-02-24 04:46:34,725][INFO ][node ] [dm-essearchp101.bldrprod.local-ElasticSearch] {elasticsearch/0.14.2}[29623]: started
[2011-02-24 04:46:39,234][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp102.bldrprod.local-ElasticSearch][myQksI6CQXup3JIGgf3iKw][inet[/10.15.104.2:9300]],}, reason: zen-disco-receive(from [[dm-essearchp104.bldrprod.local-ElasticSearch][Dd3P4wdxR5mvvkZ4D70ZMg][inet[/10.15.104.4:9300]]])
[2011-02-24 04:46:48,782][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp103.bldrprod.local-ElasticSearch][L33jpQHCSn6a8gA0jCwpJw][inet[/10.15.104.3:9300]],}, reason: zen-disco-receive(from [[dm-essearchp104.bldrprod.local-ElasticSearch][Dd3P4wdxR5mvvkZ4D70ZMg][inet[/10.15.104.4:9300]]])
[2011-02-24 04:59:54,596][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp101.bldrprod.local-ESIndexer][mJM7l5rqR0infq0FKUk56g][inet[/10.15.104.1:9301]]{client=true, data=false},}, reason: zen-disco-receive(from [[dm-essearchp104.bldrprod.local-ElasticSearch][Dd3P4wdxR5mvvkZ4D70ZMg][inet[/10.15.104.4:9300]]])
[2011-02-24 05:00:02,093][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp102.bldrprod.local-ESIndexer][hcrPj5IDRtWeVyb6Ti_ZoQ][inet[/10.15.104.2:9301]]{client=true, data=false},}, reason: zen-disco-receive(from [[dm-essearchp104.bldrprod.local-ElasticSearch][Dd3P4wdxR5mvvkZ4D70ZMg][inet[/10.15.104.4:9300]]])
[2011-02-24 05:00:11,600][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp103.bldrprod.local-ESIndexer][C8Y1x1tzQPCo9l0nKGzttA][inet[/10.15.104.3:9301]]{client=true, data=false},}, reason: zen-disco-receive(from [[dm-essearchp104.bldrprod.local-ElasticSearch][Dd3P4wdxR5mvvkZ4D70ZMg][inet[/10.15.104.4:9300]]])
[2011-02-24 05:00:19,385][INFO ][cluster.service ] [dm-essearchp101.bldrprod.local-ElasticSearch] added {[dm-essearchp104.bldrprod.local-ESIndexer][4w6xiaSbRT6u3aktprANyQ][inet[/10.15.104.4:9301]]{client=true, data=false},}, reason: zen-disco-receive(from [[dm-essearchp104.bldrprod.local-ElasticSearch][Dd3P4wdxR5mvvkZ4D70ZMg][inet[/10.15.104.4:9300]]])
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment