gistfile1.txt

./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,GROUP=P0 -t fts.moving_topology_fts.MovingTopFTS.swap_rebalance_kv_during_index_building,items=30000,cluster=D,F,replicas=0,GROUP=P0

Test Input params:
{'get-coredumps': 'True', 'logs_folder': '/data/workspace/cen006-p0-fts-vset02-00-moving-topology/logs/testrunner-16-May-05_21-56-05/test_8', 'GROUP': 'P0', 'replicas': '0', 'items': '30000', 'conf_file': 'conf/fts/py-fts-movingtopology.conf', 'num_nodes': 5, 'cluster_name': 'INI_FILE', 'cluster': 'D,F', 'stop-on-failure': 'False', 'ini': 'INI_FILE.ini', 'case_number': 8, 'get-logs': 'False', 'get-cbcollect-info': 'True', 'spec': 'py-fts-movingtopology'}
[2016-05-05 23:10:47,374] - [remote_util:169] INFO - connecting to 172.23.105.96 with username : root password : couchbase ssh_key: 
[2016-05-05 23:10:47,612] - [remote_util:203] INFO - Connected to 172.23.105.96
looking for Erlang/Memcached crashes on 172.23.105.96 ... 
dump files not found on 172.23.105.96
[2016-05-05 23:10:48,412] - [remote_util:169] INFO - connecting to 172.23.105.190 with username : root password : couchbase ssh_key: 
[2016-05-05 23:10:48,827] - [remote_util:203] INFO - Connected to 172.23.105.190
looking for Erlang/Memcached crashes on 172.23.105.190 ... 
dump files not found on 172.23.105.190
[2016-05-05 23:10:49,614] - [remote_util:169] INFO - connecting to 172.23.106.66 with username : root password : couchbase ssh_key: 
[2016-05-05 23:10:49,855] - [remote_util:203] INFO - Connected to 172.23.106.66
looking for Erlang/Memcached crashes on 172.23.106.66 ... 
dump files not found on 172.23.106.66
[2016-05-05 23:10:50,648] - [remote_util:169] INFO - connecting to 172.23.105.69 with username : root password : couchbase ssh_key: 
[2016-05-05 23:10:50,889] - [remote_util:203] INFO - Connected to 172.23.105.69
looking for Erlang/Memcached crashes on 172.23.105.69 ... 
dump files not found on 172.23.105.69
[2016-05-05 23:10:51,650] - [remote_util:169] INFO - connecting to 172.23.105.54 with username : root password : couchbase ssh_key: 
[2016-05-05 23:10:51,859] - [remote_util:203] INFO - Connected to 172.23.105.54
swap_rebalance_kv_during_index_building (fts.moving_topology_fts.MovingTopFTS) ... looking for Erlang/Memcached crashes on 172.23.105.54 ... 
dump files not found on 172.23.105.54
[2016-05-05 23:10:52,742] - [fts_base:1925] INFO - ==== FTSbasetests setup is started for test #8 swap_rebalance_kv_during_index_building ====
[2016-05-05 23:10:52,763] - [fts_base:909] INFO - removing nodes from cluster ...
[2016-05-05 23:10:52,780] - [fts_base:911] INFO - cleanup [ip:172.23.105.96 port:8091 ssh_username:root]
[2016-05-05 23:10:52,796] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.105.96
[2016-05-05 23:10:53,050] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.96:8091
[2016-05-05 23:10:53,056] - [cluster_helper:80] INFO - ns_server @ 172.23.105.96:8091 is running
[2016-05-05 23:10:53,057] - [fts_base:858] INFO - Initializing Cluster ...
[2016-05-05 23:10:53,763] - [rest_client:876] INFO - pools/default params : memoryQuota=2069
[2016-05-05 23:10:53,771] - [rest_client:904] INFO - settings/indexes params : storageMode=forestdb
[2016-05-05 23:10:53,781] - [rest_client:792] INFO - settings/web params on 172.23.105.96:8091:username=Administrator&password=password&port=8091
[2016-05-05 23:10:53,812] - [fts_base:874] INFO - 172.23.105.190 will be configured with services fts
[2016-05-05 23:10:54,813] - [task:405] INFO - adding node 172.23.105.190:8091 to cluster
[2016-05-05 23:10:54,814] - [rest_client:1062] INFO - adding remote node @172.23.105.190:8091 to this cluster @172.23.105.96:8091
[2016-05-05 23:10:59,385] - [rest_client:1280] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.105.96%2Cns_1%40172.23.105.190
[2016-05-05 23:10:59,395] - [rest_client:1284] INFO - rebalance operation started
[2016-05-05 23:10:59,403] - [rest_client:1415] INFO - rebalance percentage : 0.00 %
[2016-05-05 23:11:09,424] - [task:521] INFO - rebalancing was completed with progress: 100% in 10.028414011 sec
[2016-05-05 23:11:10,448] - [rest_client:1856] INFO - http://172.23.105.96:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1569&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=0&replicaIndex=1&saslPassword=None
[2016-05-05 23:11:10,469] - [rest_client:1878] INFO - 0.02 seconds to create bucket default
[2016-05-05 23:11:10,470] - [bucket_helper:308] INFO - waiting for memcached bucket : default in 172.23.105.96 to accept set ops
[2016-05-05 23:11:11,637] - [data_helper:295] INFO - creating direct client 172.23.105.96:11210 default
[2016-05-05 23:11:11,750] - [data_helper:295] INFO - creating direct client 172.23.105.96:11210 default
[2016-05-05 23:11:14,387] - [task:289] INFO - bucket 'default' was created with per node RAM quota: 1569
[2016-05-05 23:11:14,448] - [rest_client:1296] INFO - /diag/eval status on 172.23.105.96:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-05-05 23:11:14,451] - [remote_util:169] INFO - connecting to 172.23.105.96 with username : root password : couchbase ssh_key: 
[2016-05-05 23:11:14,696] - [remote_util:203] INFO - Connected to 172.23.105.96
[2016-05-05 23:11:15,266] - [remote_util:2292] INFO - running command.raw on 172.23.105.96: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-05-05 23:11:15,311] - [remote_util:2329] INFO - command executed successfully
[2016-05-05 23:11:15,312] - [fts_base:2362] INFO - 0
[2016-05-05 23:11:15,313] - [remote_util:2292] INFO - running command.raw on 172.23.105.96: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-05-05 23:11:15,419] - [remote_util:2329] INFO - command executed successfully
[2016-05-05 23:11:15,420] - [fts_base:2362] INFO - 0
[2016-05-05 23:11:15,536] - [remote_util:169] INFO - connecting to 172.23.105.190 with username : root password : couchbase ssh_key: 
[2016-05-05 23:11:15,781] - [remote_util:203] INFO - Connected to 172.23.105.190
[2016-05-05 23:11:16,452] - [remote_util:2292] INFO - running command.raw on 172.23.105.190: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-05-05 23:11:16,558] - [remote_util:2329] INFO - command executed successfully
[2016-05-05 23:11:16,559] - [fts_base:2362] INFO - 0
[2016-05-05 23:11:16,561] - [remote_util:2292] INFO - running command.raw on 172.23.105.190: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-05-05 23:11:16,708] - [remote_util:2329] INFO - command executed successfully
[2016-05-05 23:11:16,709] - [fts_base:2362] INFO - 0
[2016-05-05 23:11:16,719] - [remote_util:169] INFO - connecting to 172.23.106.66 with username : root password : couchbase ssh_key: 
[2016-05-05 23:11:16,978] - [remote_util:203] INFO - Connected to 172.23.106.66
[2016-05-05 23:11:17,529] - [remote_util:2292] INFO - running command.raw on 172.23.106.66: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-05-05 23:11:17,649] - [remote_util:2329] INFO - command executed successfully
[2016-05-05 23:11:17,650] - [fts_base:2362] INFO - 0
[2016-05-05 23:11:17,650] - [remote_util:2292] INFO - running command.raw on 172.23.106.66: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-05-05 23:11:17,764] - [remote_util:2329] INFO - command executed successfully
[2016-05-05 23:11:17,766] - [fts_base:2362] INFO - 0
[2016-05-05 23:11:17,883] - [remote_util:169] INFO - connecting to 172.23.105.69 with username : root password : couchbase ssh_key: 
[2016-05-05 23:11:18,125] - [remote_util:203] INFO - Connected to 172.23.105.69
[2016-05-05 23:11:18,760] - [remote_util:2292] INFO - running command.raw on 172.23.105.69: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-05-05 23:11:18,870] - [remote_util:2329] INFO - command executed successfully
[2016-05-05 23:11:18,871] - [fts_base:2362] INFO - 0
[2016-05-05 23:11:18,871] - [remote_util:2292] INFO - running command.raw on 172.23.105.69: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-05-05 23:11:18,977] - [remote_util:2329] INFO - command executed successfully
[2016-05-05 23:11:18,978] - [fts_base:2362] INFO - 0
[2016-05-05 23:11:19,095] - [remote_util:169] INFO - connecting to 172.23.105.54 with username : root password : couchbase ssh_key: 
[2016-05-05 23:11:19,307] - [remote_util:203] INFO - Connected to 172.23.105.54
[2016-05-05 23:11:19,910] - [remote_util:2292] INFO - running command.raw on 172.23.105.54: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-05-05 23:11:20,043] - [remote_util:2329] INFO - command executed successfully
[2016-05-05 23:11:20,045] - [fts_base:2362] INFO - 0
[2016-05-05 23:11:20,045] - [remote_util:2292] INFO - running command.raw on 172.23.105.54: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-05-05 23:11:20,171] - [remote_util:2329] INFO - command executed successfully
[2016-05-05 23:11:20,172] - [fts_base:2362] INFO - 0
[2016-05-05 23:11:20,174] - [fts_base:2133] INFO - {'172.23.105.190': {'crash in forestdb': 0, 'panic:': 0}, '172.23.106.66': {'crash in forestdb': 0, 'panic:': 0}, '172.23.105.69': {'crash in forestdb': 0, 'panic:': 0}, '172.23.105.54': {'crash in forestdb': 0, 'panic:': 0}, '172.23.105.96': {'crash in forestdb': 0, 'panic:': 0}}
[2016-05-05 23:11:20,174] - [fts_base:1934] INFO - ==== FTSbasetests setup is finished for test #8 swap_rebalance_kv_during_index_building ====
[2016-05-05 23:11:25,331] - [data_helper:295] INFO - creating direct client 172.23.105.96:11210 default
[2016-05-05 23:11:39,025] - [fts_base:2718] INFO - Loading phase complete!
[2016-05-05 23:11:39,056] - [fts_base:601] INFO - Checking if index already exists ...
[2016-05-05 23:11:39,079] - [rest_client:775] ERROR - http://172.23.105.190:8094/api/index/default_index_1 error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found 
 rest_auth: preparePerm, err: index not found 
[2016-05-05 23:11:39,084] - [rest_client:775] ERROR - http://172.23.105.190:8094/api/index/default_index_1 error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found 
 rest_auth: preparePerm, err: index not found 
[2016-05-05 23:11:39,084] - [fts_base:609] INFO - Creating fulltext-index default_index_1 on 172.23.105.190
[2016-05-05 23:11:39,084] - [rest_client:2158] INFO - {"params": {}, "name": "default_index_1", "planParams": {"hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {"authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0}, "uuid": ""}
[2016-05-05 23:11:39,133] - [rest_client:2167] INFO - Index default_index_1 created
[2016-05-05 23:11:39,134] - [fts_base:2472] INFO - Validating index distribution for default_index_1 ...
[2016-05-05 23:11:39,161] - [fts_base:2383] INFO - sleep for 5 secs. No pindexes found, waiting for index to get created ...
[2016-05-05 23:11:44,237] - [fts_base:2490] INFO - Validated: Number of PIndexes = 32
[2016-05-05 23:11:44,249] - [fts_base:2501] INFO - Validated: Every pIndex serves 32 partitions or lesser
[2016-05-05 23:11:44,249] - [fts_base:2517] INFO - Validated: pIndexes are distributed across [u'02ec9d8c8bbb2512252501c3b846b1ab'] 
[2016-05-05 23:11:44,250] - [fts_base:2523] INFO - Expecting num of partitions in each node in range 992-1024
[2016-05-05 23:11:44,250] - [fts_base:2538] INFO - Validated: Node 02ec9d8c8bbb2512252501c3b846b1ab houses 32 pindexes which serve 1024 partitions
[2016-05-05 23:11:44,250] - [fts_base:2383] INFO - sleep for 10 secs.  ...
[2016-05-05 23:11:54,260] - [moving_topology_fts:78] INFO - Index building has begun...
[2016-05-05 23:11:54,476] - [moving_topology_fts:81] INFO - Index count for default_index_1: 2554
[2016-05-05 23:11:54,477] - [fts_base:1671] INFO - Starting swap-rebalance [remove_node:[ip:172.23.105.96 port:8091 ssh_username:root]] -> [add_node:[ip:172.23.105.54 port:8091 ssh_username:root]] at C1 cluster 172.23.105.96
[2016-05-05 23:11:54,545] - [rest_client:1720] INFO - Node versions in cluster [u'4.5.0-2478-enterprise', u'4.5.0-2478-enterprise']
[2016-05-05 23:11:54,570] - [task:391] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2016-05-05 23:11:54,570] - [task:405] INFO - adding node 172.23.105.54:8091 to cluster
[2016-05-05 23:11:54,570] - [rest_client:1062] INFO - adding remote node @172.23.105.54:8091 to this cluster @172.23.105.96:8091
[2016-05-05 23:12:02,011] - [rest_client:1280] INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.23.105.96&user=Administrator&knownNodes=ns_1%40172.23.105.54%2Cns_1%40172.23.105.96%2Cns_1%40172.23.105.190
[2016-05-05 23:12:02,022] - [rest_client:1284] INFO - rebalance operation started
[2016-05-05 23:12:02,022] - [task:455] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2016-05-05 23:12:02,059] - [rest_client:1415] INFO - rebalance percentage : 0.00 %
[2016-05-05 23:12:12,073] - [task:455] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2016-05-05 23:12:12,120] - [rest_client:1415] INFO - rebalance percentage : 14.32 %
[2016-05-05 23:12:22,136] - [task:455] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2016-05-05 23:12:22,216] - [rest_client:1415] INFO - rebalance percentage : 32.29 %
[2016-05-05 23:12:32,230] - [task:455] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2016-05-05 23:12:32,273] - [rest_client:1415] INFO - rebalance percentage : 44.40 %
[2016-05-05 23:12:42,286] - [task:455] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2016-05-05 23:12:42,358] - [rest_client:1415] INFO - rebalance percentage : 62.76 %
[2016-05-05 23:12:52,372] - [task:455] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2016-05-05 23:12:52,418] - [rest_client:1415] INFO - rebalance percentage : 66.00 %
[2016-05-05 23:13:02,431] - [task:455] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2016-05-05 23:13:02,487] - [task:521] INFO - rebalancing was completed with progress: 100% in 60.4652690887 sec
[2016-05-05 23:13:02,488] - [fts_base:2472] INFO - Validating index distribution for default_index_1 ...
[2016-05-05 23:13:02,619] - [fts_base:2490] INFO - Validated: Number of PIndexes = 32
[2016-05-05 23:13:02,636] - [fts_base:2501] INFO - Validated: Every pIndex serves 32 partitions or lesser
[2016-05-05 23:13:02,637] - [fts_base:2517] INFO - Validated: pIndexes are distributed across [u'02ec9d8c8bbb2512252501c3b846b1ab'] 
[2016-05-05 23:13:02,637] - [fts_base:2523] INFO - Expecting num of partitions in each node in range 992-1024
[2016-05-05 23:13:02,638] - [fts_base:2538] INFO - Validated: Node 02ec9d8c8bbb2512252501c3b846b1ab houses 32 pindexes which serve 1024 partitions
[2016-05-05 23:13:02,772] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 17263
[2016-05-05 23:13:08,900] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 18542
[2016-05-05 23:13:15,034] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 20437
[2016-05-05 23:13:21,188] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 21792
[2016-05-05 23:13:27,340] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 23446
[2016-05-05 23:13:33,571] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 24438
[2016-05-05 23:13:39,675] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 25611
[2016-05-05 23:13:45,878] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 26835
[2016-05-05 23:13:52,007] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 28057
[2016-05-05 23:13:58,156] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29353
[2016-05-05 23:14:04,256] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:14:10,369] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:14:16,483] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:14:22,606] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:14:28,748] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:14:34,860] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:14:40,978] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:14:47,104] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:14:53,221] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:14:59,328] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:15:05,472] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:15:11,576] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:15:17,700] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:15:23,821] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:15:29,938] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:15:36,059] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:15:42,186] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:15:48,297] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:15:54,418] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:16:00,534] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:16:06,671] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
[2016-05-05 23:16:12,678] - [fts_base:2425] INFO - FTS indexed 29786 docs in 3.17 mins
[2016-05-05 23:16:12,790] - [fts_base:2632] INFO - Docs in index default_index_1=29786, bucket docs=30000
[2016-05-05 23:16:12,791] - [fts_base:1671] INFO - Starting swap-rebalance [remove_node:[ip:172.23.105.190 port:8091 ssh_username:root]] -> [add_node:[ip:172.23.105.69 port:8091 ssh_username:root]] at C1 cluster 172.23.105.190
[2016-05-05 23:16:13,723] - [rest_client:1720] INFO - Node versions in cluster [u'4.5.0-2478-enterprise', u'4.5.0-2478-enterprise']
[2016-05-05 23:16:13,749] - [task:405] INFO - adding node 172.23.105.69:8091 to cluster
[2016-05-05 23:16:13,749] - [rest_client:1062] INFO - adding remote node @172.23.105.69:8091 to this cluster @172.23.105.190:8091
[2016-05-05 23:16:19,578] - [rest_client:1280] INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.23.105.190&user=Administrator&knownNodes=ns_1%40172.23.105.54%2Cns_1%40172.23.105.69%2Cns_1%40172.23.105.190
[2016-05-05 23:16:19,590] - [rest_client:1284] INFO - rebalance operation started
[2016-05-05 23:16:19,598] - [rest_client:1415] INFO - rebalance percentage : 0.00 %
[2016-05-05 23:16:29,619] - [rest_client:1415] INFO - rebalance percentage : 33.00 %
[2016-05-05 23:16:39,650] - [rest_client:1415] INFO - rebalance percentage : 35.42 %
[2016-05-05 23:16:49,671] - [rest_client:1415] INFO - rebalance percentage : 35.42 %
[2016-05-05 23:16:59,691] - [rest_client:1415] INFO - rebalance percentage : 35.42 %
[2016-05-05 23:17:09,745] - [rest_client:1415] INFO - rebalance percentage : 39.58 %
[2016-05-05 23:17:19,765] - [rest_client:1415] INFO - rebalance percentage : 39.58 %
[2016-05-05 23:17:29,790] - [rest_client:1415] INFO - rebalance percentage : 39.58 %
[2016-05-05 23:17:39,814] - [rest_client:1415] INFO - rebalance percentage : 39.58 %
[2016-05-05 23:17:49,836] - [rest_client:1415] INFO - rebalance percentage : 43.75 %
[2016-05-05 23:17:59,867] - [rest_client:1415] INFO - rebalance percentage : 43.75 %
[2016-05-05 23:18:09,885] - [rest_client:1415] INFO - rebalance percentage : 43.75 %
[2016-05-05 23:18:19,919] - [rest_client:1415] INFO - rebalance percentage : 45.83 %
[2016-05-05 23:18:29,939] - [rest_client:1415] INFO - rebalance percentage : 45.83 %
[2016-05-05 23:18:39,958] - [rest_client:1415] INFO - rebalance percentage : 45.83 %
[2016-05-05 23:18:50,018] - [rest_client:1415] INFO - rebalance percentage : 47.92 %
[2016-05-05 23:19:00,040] - [rest_client:1415] INFO - rebalance percentage : 47.92 %
[2016-05-05 23:19:10,059] - [rest_client:1415] INFO - rebalance percentage : 47.92 %
[2016-05-05 23:19:20,088] - [rest_client:1415] INFO - rebalance percentage : 50.00 %
[2016-05-05 23:19:30,106] - [rest_client:1415] INFO - rebalance percentage : 50.00 %
[2016-05-05 23:19:40,129] - [rest_client:1415] INFO - rebalance percentage : 50.00 %
[2016-05-05 23:19:50,149] - [rest_client:1415] INFO - rebalance percentage : 52.08 %
[2016-05-05 23:20:00,167] - [rest_client:1415] INFO - rebalance percentage : 52.08 %
[2016-05-05 23:20:10,187] - [rest_client:1415] INFO - rebalance percentage : 52.08 %
[2016-05-05 23:20:20,219] - [rest_client:1415] INFO - rebalance percentage : 52.08 %
[2016-05-05 23:20:30,242] - [rest_client:1415] INFO - rebalance percentage : 54.17 %
[2016-05-05 23:20:40,264] - [rest_client:1415] INFO - rebalance percentage : 54.17 %
[2016-05-05 23:20:50,282] - [rest_client:1415] INFO - rebalance percentage : 54.17 %
[2016-05-05 23:21:00,302] - [rest_client:1415] INFO - rebalance percentage : 58.33 %
[2016-05-05 23:21:10,323] - [rest_client:1415] INFO - rebalance percentage : 58.33 %
[2016-05-05 23:21:20,344] - [rest_client:1415] INFO - rebalance percentage : 58.33 %
[2016-05-05 23:21:30,367] - [rest_client:1415] INFO - rebalance percentage : 68.75 %
[2016-05-05 23:21:40,385] - [rest_client:1415] INFO - rebalance percentage : 68.75 %
[2016-05-05 23:21:50,406] - [rest_client:1415] INFO - rebalance percentage : 68.75 %
[2016-05-05 23:22:00,428] - [rest_client:1415] INFO - rebalance percentage : 70.83 %
[2016-05-05 23:22:10,446] - [rest_client:1415] INFO - rebalance percentage : 70.83 %
[2016-05-05 23:22:20,464] - [rest_client:1415] INFO - rebalance percentage : 70.83 %
[2016-05-05 23:22:30,489] - [rest_client:1415] INFO - rebalance percentage : 77.08 %
[2016-05-05 23:22:40,516] - [rest_client:1415] INFO - rebalance percentage : 77.08 %
[2016-05-05 23:22:50,540] - [rest_client:1415] INFO - rebalance percentage : 77.08 %
[2016-05-05 23:23:00,559] - [rest_client:1415] INFO - rebalance percentage : 77.08 %
[2016-05-05 23:23:10,578] - [rest_client:1415] INFO - rebalance percentage : 81.25 %
[2016-05-05 23:23:20,598] - [rest_client:1415] INFO - rebalance percentage : 81.25 %
[2016-05-05 23:23:30,616] - [rest_client:1415] INFO - rebalance percentage : 81.25 %
[2016-05-05 23:23:40,636] - [rest_client:1415] INFO - rebalance percentage : 89.58 %
[2016-05-05 23:23:50,657] - [rest_client:1415] INFO - rebalance percentage : 89.58 %
[2016-05-05 23:24:00,676] - [rest_client:1415] INFO - rebalance percentage : 89.58 %
[2016-05-05 23:24:10,695] - [rest_client:1415] INFO - rebalance percentage : 93.75 %
[2016-05-05 23:24:20,720] - [rest_client:1415] INFO - rebalance percentage : 93.75 %
[2016-05-05 23:24:30,739] - [rest_client:1415] INFO - rebalance percentage : 93.75 %
[2016-05-05 23:24:40,760] - [rest_client:1415] INFO - rebalance percentage : 97.92 %
[2016-05-05 23:24:50,784] - [rest_client:1415] INFO - rebalance percentage : 97.92 %
[2016-05-05 23:25:00,803] - [rest_client:1415] INFO - rebalance percentage : 100.00 %
[2016-05-05 23:25:04,838] - [rest_client:778] ERROR - socket error while connecting to http://172.23.105.190:8091/pools error [Errno 111] Connection refused 
[2016-05-05 23:25:07,843] - [rest_client:778] ERROR - socket error while connecting to http://172.23.105.190:8091/pools error [Errno 111] Connection refused 
[2016-05-05 23:25:10,856] - [task:521] INFO - rebalancing was completed with progress: 100% in 531.265217066 sec
[2016-05-05 23:25:10,952] - [fts_base:2632] INFO - Docs in index default_index_1=30000, bucket docs=30000
FAIL