Skip to content

Instantly share code, notes, and snippets.

@itsadok

itsadok/buzzilla_data008.log Secret

Created September 10, 2014 10:15
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save itsadok/97453743d6b211681aca to your computer and use it in GitHub Desktop.
Save itsadok/97453743d6b211681aca to your computer and use it in GitHub Desktop.
Download ZIP
Data loss on network disconnect
Raw
buzzilla_data008.log
[2014-09-10 11:42:05,521][INFO ][discovery.zen ] [buzzilla_data008] master_left [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}], reason [failed to ping, tried [3] times, each with maximum [30s] timeout]
[2014-09-10 11:42:05,522][INFO ][cluster.service ] [buzzilla_data008] master {new [buzzilla_data017][239VxZmHR7m3rI0hb0CqFA][data017][inet[/10.1.2.11:9300]]{tag=gen1}, previous [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}}, removed {[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1},}, reason: zen-disco-master_failed ([buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1})
[2014-09-10 11:42:05,526][WARN ][cluster.action.shard ] [buzzilla_data008] failed to send shard started to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:42:36,909][INFO ][cluster.service ] [buzzilla_data008] detected_master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}, added {[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1},}, reason: zen-disco-receive(from master [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}])
[2014-09-10 11:43:01,776][INFO ][cluster.service ] [buzzilla_data008] master {new [buzzilla_data017][239VxZmHR7m3rI0hb0CqFA][data017][inet[/10.1.2.11:9300]]{tag=gen1}, previous [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}}, removed {[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1},}, reason: zen-disco-receive(from master [[buzzilla_data017][239VxZmHR7m3rI0hb0CqFA][data017][inet[/10.1.2.11:9300]]{tag=gen1}])
[2014-09-10 11:44:55,034][INFO ][cluster.service ] [buzzilla_data008] detected_master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}, added {[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1},}, reason: zen-disco-receive(from master [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}])
[2014-09-10 11:45:32,407][INFO ][cluster.service ] [buzzilla_data008] master {new [buzzilla_data017][239VxZmHR7m3rI0hb0CqFA][data017][inet[/10.1.2.11:9300]]{tag=gen1}, previous [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}}, removed {[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1},[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1},}, reason: zen-disco-receive(from master [[buzzilla_data017][239VxZmHR7m3rI0hb0CqFA][data017][inet[/10.1.2.11:9300]]{tag=gen1}])
[2014-09-10 11:45:32,434][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] master [[buzzilla_data017][239VxZmHR7m3rI0hb0CqFA][data017][inet[/10.1.2.11:9300]]{tag=gen1}] marked shard as started, but shard has not been created, mark shard as failed
[2014-09-10 11:45:32,434][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[STARTED], indexUUID [_na_], reason [master [buzzilla_data017][239VxZmHR7m3rI0hb0CqFA][data017][inet[/10.1.2.11:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 11:45:32,445][DEBUG][action.search.type ] [buzzilla_data008] [el-2012-10-15-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@5c5f32ea] lastShard [true]
[2014-09-10 11:45:32,446][DEBUG][action.search.type ] [buzzilla_data008] [el-2012-10-29-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@5c5f32ea] lastShard [true]
[2014-09-10 11:45:33,323][WARN ][search.action ] [buzzilla_data008] Failed to send release search context
[2014-09-10 11:45:33,324][WARN ][search.action ] [buzzilla_data008] Failed to send release search context
[2014-09-10 11:45:33,325][WARN ][search.action ] [buzzilla_data008] Failed to send release search context
[2014-09-10 11:45:33,324][WARN ][search.action ] [buzzilla_data008] Failed to send release search context
[2014-09-10 11:45:33,324][WARN ][search.action ] [buzzilla_data008] Failed to send release search context
[2014-09-10 11:45:33,324][WARN ][search.action ] [buzzilla_data008] Failed to send release search context
[2014-09-10 11:45:33,323][WARN ][search.action ] [buzzilla_data008] Failed to send release search context
[2014-09-10 11:45:33,323][WARN ][search.action ] [buzzilla_data008] Failed to send release search context
[2014-09-10 11:45:33,341][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:45:33,342][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:45:49,425][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:45:49,425][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:45:49,934][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:45:49,935][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:46:26,189][INFO ][cluster.service ] [buzzilla_data008] detected_master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}, added {[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1},[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1},}, reason: zen-disco-receive(from master [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}])
[2014-09-10 11:46:56,229][WARN ][cluster.service ] [buzzilla_data008] failed to connect to node [[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}]
[2014-09-10 11:46:56,286][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:46:56,287][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:46:56,294][INFO ][cluster.service ] [buzzilla_data008] removed {[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1},}, reason: zen-disco-receive(from master [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}])
[2014-09-10 11:46:57,481][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:46:57,482][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:14,371][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:47:14,377][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:19,135][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:47:19,135][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:24,089][INFO ][cluster.service ] [buzzilla_data008] added {[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1},}, reason: zen-disco-receive(from master [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}])
[2014-09-10 11:47:25,007][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:47:25,008][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:26,311][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:47:26,311][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:26,660][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:47:26,660][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:27,066][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:47:27,067][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:29,479][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:47:29,480][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:30,917][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:47:30,918][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:31,550][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:47:31,553][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:36,363][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:47:36,364][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:41,621][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:47:41,621][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:45,614][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:47:45,614][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:49,503][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:47:49,503][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:48:35,102][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:48:35,103][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:48:43,737][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:48:43,738][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:49:02,451][WARN ][indices.cluster ] [buzzilla_data008] [el-2011-10-31-0000][0] failed to start shard
[2014-09-10 11:49:02,452][WARN ][cluster.action.shard ] [buzzilla_data008] [el-2011-10-31-0000][0] sending failed shard for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
Raw
buzzilla_data009.log
[2014-09-10 11:42:35,836][WARN ][transport ] [buzzilla_data009] Received response for a request that has timed out, sent [137974ms] ago, timed out [11ms] ago, action [discovery/zen/fd/ping], node [[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}], id [922947339]
[2014-09-10 11:44:55,565][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[KYBJqQmaT2SSGzBblM9dtQ], [P], s[STARTED], indexUUID [_na_], reason [master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 11:45:09,918][DEBUG][action.admin.cluster.node.stats] [buzzilla_data009] failed to execute on node [dFFQV06aSA6gRmq5pAh5pg]
[2014-09-10 11:45:39,918][DEBUG][action.admin.cluster.node.stats] [buzzilla_data009] failed to execute on node [dFFQV06aSA6gRmq5pAh5pg]
[2014-09-10 11:46:09,918][DEBUG][action.admin.cluster.node.stats] [buzzilla_data009] failed to execute on node [dFFQV06aSA6gRmq5pAh5pg]
[2014-09-10 11:46:39,918][DEBUG][action.admin.cluster.node.stats] [buzzilla_data009] failed to execute on node [dFFQV06aSA6gRmq5pAh5pg]
[2014-09-10 11:46:47,190][WARN ][transport ] [buzzilla_data009] Received response for a request that has timed out, sent [111345ms] ago, timed out [81345ms] ago, action [discovery/zen/fd/ping], node [[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}], id [922949468]
[2014-09-10 11:46:47,191][WARN ][transport ] [buzzilla_data009] Received response for a request that has timed out, sent [81345ms] ago, timed out [51345ms] ago, action [discovery/zen/fd/ping], node [[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}], id [922951268]
[2014-09-10 11:46:47,191][WARN ][transport ] [buzzilla_data009] Received response for a request that has timed out, sent [51346ms] ago, timed out [21345ms] ago, action [discovery/zen/fd/ping], node [[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}], id [922952839]
[2014-09-10 11:46:47,209][WARN ][transport ] [buzzilla_data009] Received response for a request that has timed out, sent [82291ms] ago, timed out [67291ms] ago, action [cluster/nodes/stats/n], node [[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}], id [922950947]
[2014-09-10 11:46:47,209][WARN ][transport ] [buzzilla_data009] Received response for a request that has timed out, sent [52291ms] ago, timed out [37291ms] ago, action [cluster/nodes/stats/n], node [[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}], id [922952534]
[2014-09-10 11:46:47,209][WARN ][transport ] [buzzilla_data009] Received response for a request that has timed out, sent [112291ms] ago, timed out [97291ms] ago, action [cluster/nodes/stats/n], node [[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}], id [922949057]
[2014-09-10 11:46:47,210][WARN ][transport ] [buzzilla_data009] Received response for a request that has timed out, sent [22292ms] ago, timed out [7292ms] ago, action [cluster/nodes/stats/n], node [[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}], id [922953432]
[2014-09-10 11:46:47,218][WARN ][transport ] [buzzilla_data009] Received response for a request that has timed out, sent [52129ms] ago, timed out [22129ms] ago, action [/gateway/local/started-shards/n], node [[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}], id [922952821]
[2014-09-10 11:46:55,368][INFO ][cluster.service ] [buzzilla_data009] removed {[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1},}, reason: zen-disco-node_failed([buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}), reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2014-09-10 11:46:56,289][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:46:57,484][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:14,379][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:19,138][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:24,080][INFO ][cluster.service ] [buzzilla_data009] added {[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1},}, reason: zen-disco-receive(join from node[[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}])
[2014-09-10 11:47:25,010][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:26,028][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2013-07-08-0000][0] received shard failed for [el-2013-07-08-0000][0], node[239VxZmHR7m3rI0hb0CqFA], [P], s[STARTED], indexUUID [_na_], reason [master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 11:47:26,029][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2012-02-27-0000][0] received shard failed for [el-2012-02-27-0000][0], node[239VxZmHR7m3rI0hb0CqFA], [P], s[STARTED], indexUUID [_na_], reason [master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 11:47:26,029][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2013-04-22-0000][0] received shard failed for [el-2013-04-22-0000][0], node[239VxZmHR7m3rI0hb0CqFA], [P], s[STARTED], indexUUID [_na_], reason [master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 11:47:26,030][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2012-07-30-0000][0] received shard failed for [el-2012-07-30-0000][0], node[239VxZmHR7m3rI0hb0CqFA], [P], s[STARTED], indexUUID [_na_], reason [master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 11:47:26,030][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2012-12-24-0000][0] received shard failed for [el-2012-12-24-0000][0], node[239VxZmHR7m3rI0hb0CqFA], [P], s[STARTED], indexUUID [_na_], reason [master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 11:47:26,314][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:26,663][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:27,069][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:28,975][WARN ][discovery.zen ] [buzzilla_data009] received a join request for an existing node [[buzzilla_data017][239VxZmHR7m3rI0hb0CqFA][data017][inet[/10.1.2.11:9300]]{tag=gen1}]
[2014-09-10 11:47:29,482][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:30,920][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:31,555][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:36,366][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:41,624][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:45,617][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:47:49,506][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:48:35,105][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:48:43,740][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:49:02,454][WARN ][cluster.action.shard ] [buzzilla_data009] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
Raw
buzzilla_data017.log
[2014-09-10 11:42:05,715][INFO ][discovery.zen ] [buzzilla_data017] master_left [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}], reason [failed to ping, tried [3] times, each with maximum [30s] timeout]
[2014-09-10 11:42:05,747][INFO ][cluster.service ] [buzzilla_data017] master {new [buzzilla_data017][239VxZmHR7m3rI0hb0CqFA][data017][inet[/10.1.2.11:9300]]{tag=gen1}, previous [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}}, removed {[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1},}, reason: zen-disco-master_failed ([buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1})
[2014-09-10 11:42:35,790][WARN ][transport ] [buzzilla_data017] Received response for a request that has timed out, sent [120079ms] ago, timed out [90079ms] ago, action [discovery/zen/fd/masterPing], node [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}], id [13696023]
[2014-09-10 11:42:35,790][WARN ][transport ] [buzzilla_data017] Received response for a request that has timed out, sent [90076ms] ago, timed out [60076ms] ago, action [discovery/zen/fd/masterPing], node [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}], id [13696024]
[2014-09-10 11:42:35,790][WARN ][transport ] [buzzilla_data017] Received response for a request that has timed out, sent [60076ms] ago, timed out [30076ms] ago, action [discovery/zen/fd/masterPing], node [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}], id [13696025]
[2014-09-10 11:42:35,800][WARN ][cluster.action.shard ] [buzzilla_data017] failed to send shard started to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:42:35,800][WARN ][cluster.action.shard ] [buzzilla_data017] failed to send shard started to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:42:50,829][DEBUG][action.admin.cluster.node.stats] [buzzilla_data017] failed to execute on node [dFFQV06aSA6gRmq5pAh5pg]
[2014-09-10 11:43:01,479][WARN ][transport ] [buzzilla_data017] Received response for a request that has timed out, sent [54718ms] ago, timed out [24718ms] ago, action [discovery/zen/fd/ping], node [[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}], id [13696050]
[2014-09-10 11:43:01,483][WARN ][transport ] [buzzilla_data017] Received response for a request that has timed out, sent [25654ms] ago, timed out [10654ms] ago, action [cluster/nodes/stats/n], node [[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}], id [13696563]
[2014-09-10 11:43:31,677][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:43:31,679][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:44:20,801][DEBUG][action.admin.cluster.node.stats] [buzzilla_data017] failed to execute on node [dFFQV06aSA6gRmq5pAh5pg]
[2014-09-10 11:44:50,801][DEBUG][action.admin.cluster.node.stats] [buzzilla_data017] failed to execute on node [dFFQV06aSA6gRmq5pAh5pg]
[2014-09-10 11:45:13,911][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:45:13,911][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:45:20,801][DEBUG][action.admin.cluster.node.stats] [buzzilla_data017] failed to execute on node [dFFQV06aSA6gRmq5pAh5pg]
[2014-09-10 11:45:25,088][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:45:25,088][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:45:32,396][INFO ][cluster.service ] [buzzilla_data017] removed {[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1},}, reason: zen-disco-node_failed([buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1}), reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2014-09-10 11:45:32,435][WARN ][cluster.action.shard ] [buzzilla_data017] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[STARTED], indexUUID [_na_], reason [master [buzzilla_data017][239VxZmHR7m3rI0hb0CqFA][data017][inet[/10.1.2.11:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 11:45:32,473][WARN ][cluster.action.shard ] [buzzilla_data017] [el-2012-10-01-0000][0] received shard failed for [el-2012-10-01-0000][0], node[Do8KzUFURP-eUCwEdJM8bA], [R], s[STARTED], indexUUID [K80LdsV7QMq_HLuFD0yy1w], reason [master [buzzilla_data017][239VxZmHR7m3rI0hb0CqFA][data017][inet[/10.1.2.11:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 11:45:32,871][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-05-06-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,871][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-07-01-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [O3HxIj9nSmiA85R0Pnt34g], [P], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,874][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2014-07-07-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,874][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2012-10-29-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,874][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-11-11-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,875][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-07-15-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,875][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2012-10-15-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,874][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2012-06-25-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [ds3iguDeTCa7GnfP2lCmlA], [P], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,874][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2014-01-20-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,874][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-05-27-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,874][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2012-10-15-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,875][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-07-15-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,875][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2012-10-15-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,875][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2012-01-02-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [RAR26zfuTiKl4mdbRVTtNA], [P], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,875][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-07-15-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,876][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2014-05-12-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [mqdnUEcyR3ecxX4MmmvDZw], [R], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,875][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-10-14-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [QTf4UCl_SluI43zypJBk0A], [P], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,875][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2014-07-07-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,875][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2014-07-07-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,877][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-10-14-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [QTf4UCl_SluI43zypJBk0A], [P], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,877][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-10-14-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [QTf4UCl_SluI43zypJBk0A], [P], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,877][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2014-05-12-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [mqdnUEcyR3ecxX4MmmvDZw], [R], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,877][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2012-10-29-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,877][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-05-06-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,877][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-rolling-archive2][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [lsarbqE_Rleiq97pVyE4Zw], [R], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,877][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-11-11-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,877][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-05-06-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,877][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-05-27-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,876][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-11-11-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,876][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2012-01-02-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [RAR26zfuTiKl4mdbRVTtNA], [P], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,876][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-05-27-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,877][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2012-06-25-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [ds3iguDeTCa7GnfP2lCmlA], [P], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,877][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2012-01-02-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [RAR26zfuTiKl4mdbRVTtNA], [P], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,878][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-rolling-archive2][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [lsarbqE_Rleiq97pVyE4Zw], [R], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,878][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-12-02-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [8v5CM28nRh222TrWBWb7sg], [R], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,878][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2012-10-29-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,878][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-12-02-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [8v5CM28nRh222TrWBWb7sg], [R], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,878][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-07-01-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [O3HxIj9nSmiA85R0Pnt34g], [P], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,878][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-rolling-archive2][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [lsarbqE_Rleiq97pVyE4Zw], [R], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,878][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-07-01-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [O3HxIj9nSmiA85R0Pnt34g], [P], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,878][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2013-12-02-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [8v5CM28nRh222TrWBWb7sg], [R], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15a070a5]
[2014-09-10 11:45:32,878][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2012-06-25-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], relocating [ds3iguDeTCa7GnfP2lCmlA], [P], s[RELOCATING]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:32,878][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2014-01-20-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@2502302]
[2014-09-10 11:45:32,878][DEBUG][action.admin.indices.stats] [buzzilla_data017] [el-2014-01-20-0000][0], node[dFFQV06aSA6gRmq5pAh5pg], [R], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@1472e1f3]
[2014-09-10 11:45:33,342][WARN ][cluster.action.shard ] [buzzilla_data017] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:45:49,426][WARN ][cluster.action.shard ] [buzzilla_data017] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:45:49,935][WARN ][cluster.action.shard ] [buzzilla_data017] [el-2011-10-31-0000][0] received shard failed for [el-2011-10-31-0000][0], node[RAR26zfuTiKl4mdbRVTtNA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[el-2011-10-31-0000][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/home/omgili/data/elasticsearch/data/buzzilla/nodes/0/indices/el-2011-10-31-0000/0/index), type=MERGE, rate=20.0)]): files: []]; ]]
[2014-09-10 11:46:25,191][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:46:25,191][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:46:55,376][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:46:55,376][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:46:57,327][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:46:57,327][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:46:57,455][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:46:57,455][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:46:57,597][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:46:57,597][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:47:14,419][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:47:14,420][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:47:14,567][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:47:14,567][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:47:19,106][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:47:19,106][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:47:19,309][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:47:19,309][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:47:24,089][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:47:24,089][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:47:24,981][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with an older cluster_state, telling [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] to rejoin the cluster
[2014-09-10 11:47:24,981][WARN ][discovery.zen ] [buzzilla_data017] failed to send rejoin request to [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:47:25,367][WARN ][discovery.zen ] [buzzilla_data017] received cluster state from [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] which is also master but with a newer cluster_state, rejoining to cluster...
[2014-09-10 11:47:25,368][WARN ][discovery.zen ] [buzzilla_data017] zen-disco-master_receive_cluster_state_from_another_master [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}], current nodes: {[buzzilla_data015][VLUub2ONQF6O8CrRyRBICQ][data015][inet[/10.10.5.149:9300]]{tag=gen1},[buzzilla_data001][PZmO9KRMS_21ud6CeeGgbw][data001][inet[/10.10.5.138:9300]]{tag=gen1},[buzzilla_data005][OwpH3oggTdWhv1vyiVWz9g][data005][inet[/10.10.5.142:9300]]{tag=gen1},[buzzilla_data013][O3HxIj9nSmiA85R0Pnt34g][data013][inet[/10.10.3.62:9300]]{tag=gen1},[buzzilla_data007][Do8KzUFURP-eUCwEdJM8bA][data007][inet[/10.1.1.10:9300]]{tag=gen1},[buzzilla_data016][SYZ2-ARWS6-7cOpJgS2nsA][data016][inet[/10.1.2.10:9300]]{tag=gen1},[buzzilla_data020][lsarbqE_Rleiq97pVyE4Zw][data020][inet[/10.10.1.71:9300]]{tag=gen1},[buzzilla_data011][wuy72260Q9KJA59o9z7lwg][data011][inet[/10.10.2.173:9300]]{tag=gen1},[buzzilla_data014][mqdnUEcyR3ecxX4MmmvDZw][data014][inet[/10.10.5.148:9300]]{tag=gen1},[buzzilla_data006][YhXsUG37SfSaSBkI470lpg][data006][inet[/10.10.5.143:9300]]{tag=gen1},[buzzilla_data018][jhjVnypiRjO4i7oplDDjMg][data018][inet[/10.1.2.12:9300]]{tag=gen1},[buzzilla_data008][RAR26zfuTiKl4mdbRVTtNA][data008][inet[/10.1.1.8:9300]]{tag=gen1},[buzzilla_data004][ds3iguDeTCa7GnfP2lCmlA][data004][inet[/10.10.5.141:9300]]{tag=gen1},[buzzilla_data021][KYBJqQmaT2SSGzBblM9dtQ][data021][inet[/10.10.1.72:9300]]{tag=gen1},[buzzilla_data019][QTf4UCl_SluI43zypJBk0A][data019][inet[/10.1.2.13:9300]]{tag=gen1},[buzzilla_data012][Uc4gdbesQ22n_KdW-3m_FA][data012][inet[/10.10.2.172:9300]]{tag=gen1},[buzzilla_data003][8v5CM28nRh222TrWBWb7sg][data003][inet[/10.10.5.140:9300]]{tag=gen1},[buzzilla_data017][239VxZmHR7m3rI0hb0CqFA][data017][inet[/10.1.2.11:9300]]{tag=gen1},}
[2014-09-10 11:47:25,368][INFO ][cluster.service ] [buzzilla_data017] removed {[buzzilla_data015][VLUub2ONQF6O8CrRyRBICQ][data015][inet[/10.10.5.149:9300]]{tag=gen1},[buzzilla_data001][PZmO9KRMS_21ud6CeeGgbw][data001][inet[/10.10.5.138:9300]]{tag=gen1},[buzzilla_data005][OwpH3oggTdWhv1vyiVWz9g][data005][inet[/10.10.5.142:9300]]{tag=gen1},[buzzilla_data013][O3HxIj9nSmiA85R0Pnt34g][data013][inet[/10.10.3.62:9300]]{tag=gen1},[buzzilla_data007][Do8KzUFURP-eUCwEdJM8bA][data007][inet[/10.1.1.10:9300]]{tag=gen1},[buzzilla_data016][SYZ2-ARWS6-7cOpJgS2nsA][data016][inet[/10.1.2.10:9300]]{tag=gen1},[buzzilla_data020][lsarbqE_Rleiq97pVyE4Zw][data020][inet[/10.10.1.71:9300]]{tag=gen1},[buzzilla_data011][wuy72260Q9KJA59o9z7lwg][data011][inet[/10.10.2.173:9300]]{tag=gen1},[buzzilla_data014][mqdnUEcyR3ecxX4MmmvDZw][data014][inet[/10.10.5.148:9300]]{tag=gen1},[buzzilla_data006][YhXsUG37SfSaSBkI470lpg][data006][inet[/10.10.5.143:9300]]{tag=gen1},[buzzilla_data018][jhjVnypiRjO4i7oplDDjMg][data018][inet[/10.1.2.12:9300]]{tag=gen1},[buzzilla_data008][RAR26zfuTiKl4mdbRVTtNA][data008][inet[/10.1.1.8:9300]]{tag=gen1},[buzzilla_data004][ds3iguDeTCa7GnfP2lCmlA][data004][inet[/10.10.5.141:9300]]{tag=gen1},[buzzilla_data021][KYBJqQmaT2SSGzBblM9dtQ][data021][inet[/10.10.1.72:9300]]{tag=gen1},[buzzilla_data019][QTf4UCl_SluI43zypJBk0A][data019][inet[/10.1.2.13:9300]]{tag=gen1},[buzzilla_data012][Uc4gdbesQ22n_KdW-3m_FA][data012][inet[/10.10.2.172:9300]]{tag=gen1},[buzzilla_data003][8v5CM28nRh222TrWBWb7sg][data003][inet[/10.10.5.140:9300]]{tag=gen1},}, reason: zen-disco-master_receive_cluster_state_from_another_master [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}]
[2014-09-10 11:47:25,841][INFO ][cluster.service ] [buzzilla_data017] detected_master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}, added {[buzzilla_data015][VLUub2ONQF6O8CrRyRBICQ][data015][inet[/10.10.5.149:9300]]{tag=gen1},[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1},[buzzilla_data001][PZmO9KRMS_21ud6CeeGgbw][data001][inet[/10.10.5.138:9300]]{tag=gen1},[buzzilla_data005][OwpH3oggTdWhv1vyiVWz9g][data005][inet[/10.10.5.142:9300]]{tag=gen1},[buzzilla_data013][O3HxIj9nSmiA85R0Pnt34g][data013][inet[/10.10.3.62:9300]]{tag=gen1},[buzzilla_data002][dFFQV06aSA6gRmq5pAh5pg][data002][inet[/10.10.5.139:9300]]{tag=gen1},[buzzilla_data007][Do8KzUFURP-eUCwEdJM8bA][data007][inet[/10.1.1.10:9300]]{tag=gen1},[buzzilla_data016][SYZ2-ARWS6-7cOpJgS2nsA][data016][inet[/10.1.2.10:9300]]{tag=gen1},[buzzilla_data020][lsarbqE_Rleiq97pVyE4Zw][data020][inet[/10.10.1.71:9300]]{tag=gen1},[buzzilla_data011][wuy72260Q9KJA59o9z7lwg][data011][inet[/10.10.2.173:9300]]{tag=gen1},[buzzilla_data014][mqdnUEcyR3ecxX4MmmvDZw][data014][inet[/10.10.5.148:9300]]{tag=gen1},[buzzilla_data006][YhXsUG37SfSaSBkI470lpg][data006][inet[/10.10.5.143:9300]]{tag=gen1},[buzzilla_data018][jhjVnypiRjO4i7oplDDjMg][data018][inet[/10.1.2.12:9300]]{tag=gen1},[buzzilla_data008][RAR26zfuTiKl4mdbRVTtNA][data008][inet[/10.1.1.8:9300]]{tag=gen1},[buzzilla_data004][ds3iguDeTCa7GnfP2lCmlA][data004][inet[/10.10.5.141:9300]]{tag=gen1},[buzzilla_data021][KYBJqQmaT2SSGzBblM9dtQ][data021][inet[/10.10.1.72:9300]]{tag=gen1},[buzzilla_data019][QTf4UCl_SluI43zypJBk0A][data019][inet[/10.1.2.13:9300]]{tag=gen1},[buzzilla_data012][Uc4gdbesQ22n_KdW-3m_FA][data012][inet[/10.10.2.172:9300]]{tag=gen1},[buzzilla_data003][8v5CM28nRh222TrWBWb7sg][data003][inet[/10.10.5.140:9300]]{tag=gen1},}, reason: zen-disco-receive(from master [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}])
[2014-09-10 11:47:26,026][WARN ][indices.cluster ] [buzzilla_data017] [el-2013-07-08-0000][0] master [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] marked shard as started, but shard has not been created, mark shard as failed
[2014-09-10 11:47:26,026][WARN ][cluster.action.shard ] [buzzilla_data017] [el-2013-07-08-0000][0] sending failed shard for [el-2013-07-08-0000][0], node[239VxZmHR7m3rI0hb0CqFA], [P], s[STARTED], indexUUID [_na_], reason [master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 11:47:26,026][WARN ][indices.cluster ] [buzzilla_data017] [el-2012-02-27-0000][0] master [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] marked shard as started, but shard has not been created, mark shard as failed
[2014-09-10 11:47:26,026][WARN ][cluster.action.shard ] [buzzilla_data017] [el-2012-02-27-0000][0] sending failed shard for [el-2012-02-27-0000][0], node[239VxZmHR7m3rI0hb0CqFA], [P], s[STARTED], indexUUID [_na_], reason [master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 11:47:26,026][WARN ][indices.cluster ] [buzzilla_data017] [el-2013-04-22-0000][0] master [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] marked shard as started, but shard has not been created, mark shard as failed
[2014-09-10 11:47:26,026][WARN ][cluster.action.shard ] [buzzilla_data017] [el-2013-04-22-0000][0] sending failed shard for [el-2013-04-22-0000][0], node[239VxZmHR7m3rI0hb0CqFA], [P], s[STARTED], indexUUID [_na_], reason [master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 11:47:26,026][WARN ][indices.cluster ] [buzzilla_data017] [el-2012-12-24-0000][0] master [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] marked shard as started, but shard has not been created, mark shard as failed
[2014-09-10 11:47:26,027][WARN ][cluster.action.shard ] [buzzilla_data017] [el-2012-12-24-0000][0] sending failed shard for [el-2012-12-24-0000][0], node[239VxZmHR7m3rI0hb0CqFA], [P], s[STARTED], indexUUID [_na_], reason [master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
[2014-09-10 11:47:26,027][WARN ][indices.cluster ] [buzzilla_data017] [el-2012-07-30-0000][0] master [[buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1}] marked shard as started, but shard has not been created, mark shard as failed
[2014-09-10 11:47:26,027][WARN ][cluster.action.shard ] [buzzilla_data017] [el-2012-07-30-0000][0] sending failed shard for [el-2012-07-30-0000][0], node[239VxZmHR7m3rI0hb0CqFA], [P], s[STARTED], indexUUID [_na_], reason [master [buzzilla_data009][1V_GNg4zRLaGfCULHtnPxQ][data009][inet[/10.1.1.9:9300]]{tag=gen1} marked shard as started, but shard has not been created, mark shard as failed]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment