tleyden/gist:31a233acf78ad60240058d6ffd09d920 Secret

## gistfile1.txt

TASK [SG ACCEL | Wait until sg_accel to listen on port] ************************
ok: [ac1]

TASK [debug] *******************************************************************
skipping: [ac1]

TASK [SG ACCEL | Start sg_accel service on Windows] ****************************
skipping: [ac1]

PLAY RECAP *********************************************************************
ac1                        : ok=8    changed=3    unreachable=0    failed=0

2017-05-08 19:49:14,927 - INFO - get_cbgt_config - GET http://192.168.33.14:4985/_cbgt/api/cfg
INFO:test_framework:GET http://192.168.33.14:4985/_cbgt/api/cfg
CBGT node to pindex counts: {u'2568051445c6ecf9': 16}
>>> Detected valid CBGT Pindex distribution
{u'load_generators': [], u'couchbase_servers': [{u'ip': u'192.168.33.10', u'name': u'cb1'}, {u'ip': u'192.168.33.11', u'name': u'cb2'}, {u'ip': u'192.168.33.12', u'name': u'cb3'}], u'sync_gateways': [{u'ip': u'192.168.33.13', u'name': u'sg1'}], u'hosts': [{u'ip': u'192.168.33.10', u'name': u'host1'}, {u'ip': u'192.168.33.11', u'name': u'host2'}, {u'ip': u'192.168.33.12', u'name': u'host3'}, {u'ip': u'192.168.33.13', u'name': u'host4'}, {u'ip': u'192.168.33.14', u'name': u'host5'}], u'load_balancers': [], u'sg_accels': [{u'ip': u'192.168.33.14', u'name': u'ac1'}], u'cbs_ssl_enabled': False}
Running: 'test_server_goes_down_sanity'
cluster_config: resources/cluster_configs/multiple_servers_di
admin_sg: http://192.168.33.13:4985
sg_url: http://192.168.33.13:4984
cbs_one_url: http://192.168.33.10:8091
cbs_two_url: http://192.168.33.11:8091
POST http://192.168.33.13:4985/db/_user/ 201
POST http://192.168.33.13:4985/db/_user/ 201
POST http://192.168.33.13:4985/db/_session 200
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 200
Found 1 changes
PUT 100 docs to http://192.168.33.13:4984/db/ with prefix None
Added: 100 docs
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 200
Found 84 changes
Missing expected docs: 17
last_seq: 89-0
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': u'89-0', 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 200
Found 17 changes
Missing expected docs: 0
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': u'1-0', 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 200
Found 100 changes
Connecting to 192.168.33.11
Running 'sudo service couchbase-server stop' on host 192.168.33.11
Stopping couchbase-server (via systemctl):                 [  OK  ]

Closing connection to 192.168.33.11
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 503
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 503
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 503
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 503
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 503
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 503
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 503
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 503
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 503
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
  > Verify http://192.168.33.13:4984/db has 100 docs in changes
GET http://192.168.33.13:4984/ 200
Using POST data: {'feed': 'longpoll', 'since': 0, 'timeout': 60000}
POST http://192.168.33.13:4984/db/_changes 404
/db/_changes failed due to server down. Retrying ...
Connecting to 192.168.33.11
Running 'sudo service couchbase-server start' on host 192.168.33.11
Starting couchbase-server (via systemctl):                 [  OK  ]

Closing connection to 192.168.33.11
GET http://192.168.33.11:8091/pools/nodes 200
All nodes are healthy
Setting recover mode to 'delta' for server 192.168.33.11
POST http://192.168.33.10:8091/controller/setRecoveryType 200
Starting rebalance in for <keywords.couchbaseserver.CouchbaseServer instance at 0x10346e908>
Known nodes: knownNodes=ns_1@192.168.33.10,ns_1@192.168.33.12,ns_1@192.168.33.11
POST http://192.168.33.10:8091/controller/rebalance 200
GET http://192.168.33.10:8091/pools/default/tasks 200
Rebalance found in tasks!
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
warming_up is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
indexer is running
indexer is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is running
GET http://192.168.33.10:8091/pools/default/tasks 200
rebalance is notRunning
view_compaction is running
FAILEDTearing down test 'test_server_goes_down_rebuild_channels'
----------------
Socket usage on mobile-testkit client ...
ESTABLISHED: 83
TIME_WAIT: 0

resources/cluster_configs/multiple_servers_di
cbs: {u'ip': u'192.168.33.12', u'name': u'cb3'}
sgs: [{'ip': u'192.168.33.13', 'name': u'sg1'}]
acs: [{'ip': u'192.168.33.14', 'name': u'ac1'}]
ssl: False
sync_gateway: http://192.168.33.13:4984, info: {"couchdb":"Welcome","vendor":{"name":"Couchbase Sync Gateway","version":1.4},"version":"Couchbase Sync Gateway/HEAD(nobranch)(5e5fd9c)"}
sg_accel: http://192.168.33.14:4985, info: {"ADMIN":true,"couchdb":"Welcome","vendor":{"name":"Couchbase Sync Gateway","version":1.4},"version":"Couchbase Sync Gateway/HEAD(nobranch)(5e5fd9c)"}
Pulling sync_gateway / sg_accel logs
Running playbook: libraries/provision/ansible/playbooks/fetch-sync-gateway-logs.yml

PLAY [sync_gateways] ***********************************************************

TASK [setup] *******************************************************************
ok: [sg1]

TASK [SYNC GATEWAY | Check if sync_gateway access log is present] **************
ok: [sg1]

TASK [SYNC GATEWAY | Fetch sync_gateway access logs] ***************************
changed: [sg1]

TASK [SYNC GATEWAY | Check if sync_gateway error log is present] ***************
ok: [sg1]

TASK [SYNC GATEWAY | Fetch sync_gateway error logs] ****************************
changed: [sg1]

TASK [SYNC GATEWAY | Check if sync_gateway access log is present on Windows] ***
skipping: [sg1]

TASK [SYNC GATEWAY | Fetch sync_gateway access log for Windows] ****************
skipping: [sg1]

TASK [SYNC GATEWAY | Check if sync_gateway error log is present on Windows] ****
skipping: [sg1]

TASK [SYNC GATEWAY | Fetch sync_gateway error log for Windows] *****************
skipping: [sg1]

PLAY [sg_accels] ***************************************************************

TASK [setup] *******************************************************************
ok: [ac1]

TASK [SG ACCEL | Check if sg_accel access log is present] **********************
ok: [ac1]

TASK [SG ACCEL | Fetch sg_accel access logs] ***********************************
changed: [ac1]

TASK [SG ACCEL | Check if sg_accel error log is present] ***********************
ok: [ac1]

TASK [SG ACCEL | Fetch sg_accel error logs] ************************************
changed: [ac1]

TASK [SG ACCEL | Check if sg_accel access logs are present] ********************
skipping: [ac1]

TASK [SG ACCEL | Fetch sg_accel access logs] ***********************************
skipping: [ac1]

TASK [SG ACCEL | Check if sg_accel error logs are present] *********************
skipping: [ac1]

TASK [SG ACCEL | Fetch sg_accel error logs] ************************************
skipping: [ac1]

PLAY RECAP *********************************************************************
ac1                        : ok=5    changed=2    unreachable=0    failed=0
sg1                        : ok=5    changed=2    unreachable=0    failed=0

sync_gateway logs copied to results/logs/test_server_goes_down_rebuild_channels-2017-05-08-19-52-53-sglogs.zip
TODO: Running Analysis: results/logs/test_server_goes_down_rebuild_channels-2017-05-08-19-52-53-sglogs.zip
Tearing down 'params_from_base_suite_setup' ...


---------------------------------------------------------------------- generated xml file: /Users/tleyden/Development/mobile-testkit/results/results.xml ----------------------------------------------------------------------
--------------------------------------------------------------------- generated html file: /Users/tleyden/Development/mobile-testkit/results/report.html ----------------------------------------------------------------------
========================================================================================================== FAILURES ===========================================================================================================
___________________________________________________________________________________________ test_server_goes_down_rebuild_channels ____________________________________________________________________________________________

params_from_base_test_setup = {'cluster_config': 'resources/cluster_configs/multiple_servers_di', 'mode': 'di'}

    @pytest.mark.sanity
    @pytest.mark.syncgateway
    @pytest.mark.changes
    @pytest.mark.failover
    @pytest.mark.session
    @pytest.mark.channel
    def test_server_goes_down_rebuild_channels(params_from_base_test_setup):
        """
        1. Start with a two node couchbase server cluster
        2. Starting adding docs
        3. Kill one of the server nodes and signal completion
        4. Stop adding docs
        5. Verify that that the expected docs are present and in the changes feed.
        6. Start server again and add to cluster
        """

        cluster_config = params_from_base_test_setup["cluster_config"]
        mode = params_from_base_test_setup["mode"]

        cluster_helper = ClusterKeywords()

        sg_conf_name = "sync_gateway_default_functional_tests"
        sg_conf_path = sync_gateway_config_path_for_mode(sg_conf_name, mode)

        cluster_helper.reset_cluster(cluster_config=cluster_config,
                                     sync_gateway_config=sg_conf_path)

        topology = cluster_helper.get_cluster_topology(cluster_config)

        admin_sg = topology["sync_gateways"][0]["admin"]
        sg_url = topology["sync_gateways"][0]["public"]
        coucbase_servers = topology["couchbase_servers"]

        cbs_one_url = coucbase_servers[0]
        cbs_two_url = coucbase_servers[1]

        log_info("Running: 'test_server_goes_down_sanity'")
        log_info("cluster_config: {}".format(cluster_config))
        log_info("admin_sg: {}".format(admin_sg))
        log_info("sg_url: {}".format(sg_url))
        log_info("cbs_one_url: {}".format(cbs_one_url))
        log_info("cbs_two_url: {}".format(cbs_two_url))

        sg_db = "db"
        num_docs = 100

        admin_user_info = userinfo.UserInfo(
            name="admin",
            password="password",
            channels=["ABC"],
            roles=[]
        )

        seth_user_info = userinfo.UserInfo(
            name="seth",
            password="password",
            channels=["ABC"],
            roles=[]
        )

        client = MobileRestClient()
        main_server = couchbaseserver.CouchbaseServer(cbs_one_url)
        flakey_server = couchbaseserver.CouchbaseServer(cbs_two_url)

        admin_auth = client.create_user(
            admin_sg,
            sg_db,
            admin_user_info.name,
            admin_user_info.password,
            channels=admin_user_info.channels
        )

        client.create_user(
            admin_sg,
            sg_db,
            seth_user_info.name,
            seth_user_info.password,
            channels=seth_user_info.channels
        )
        seth_session = client.create_session(admin_sg, sg_db, seth_user_info.name)

        # allow any user docs to make it to changes
        initial_changes = client.get_changes(url=sg_url, db=sg_db, since=0, auth=seth_session)

        # push docs from admin
        docs = client.add_docs(
            url=sg_url,
            db=sg_db,
            number=num_docs,
            id_prefix=None,
            channels=admin_user_info.channels,
            auth=admin_auth
        )

        assert len(docs) == num_docs

        client.verify_docs_in_changes(url=sg_url, db=sg_db, expected_docs=docs, auth=seth_session)
        changes_before_failover = client.get_changes(url=sg_url, db=sg_db, since=initial_changes["last_seq"], auth=seth_session)
        assert len(changes_before_failover["results"]) == num_docs

        # Stop server via 'service stop'
        flakey_server.stop()

        start = time.time()
        while True:
            # Fail tests if all docs do not succeed before timeout
            if (time.time() - start) > 60:
                # Bring server back up before failing the test
                flakey_server.start()
                main_server.recover(flakey_server)
                main_server.rebalance_in(coucbase_servers, flakey_server)
>               raise keywords.exceptions.TimeoutError("Failed to rebuild changes")
E               TimeoutError: Failed to rebuild changes

testsuites/syncgateway/functional/topology_specific_tests/multiple_servers/test_multiple_servers.py:321: TimeoutError
===================================================================================================== 22 tests deselected =====================================================================================================
========================================================================================== 1 failed, 22 deselected in 257.36 seconds ==============================