Skip to content

Instantly share code, notes, and snippets.

@arunapiravi

arunapiravi/gist:c39e0677742882057365ee8049d89234

Created August 27, 2018 02:12
Show Gist options
  • Save arunapiravi/c39e0677742882057365ee8049d89234 to your computer and use it in GitHub Desktop.
Save arunapiravi/c39e0677742882057365ee8049d89234 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
./testrunner -i /tmp/testexec.14385.ini -p get-cbcollect-info=True,disable_HTP=True,get-logs=False,stop-on-failure=False,fts_quota=512,upgrade_version=6.0.0-1564 -t upgrade.upgrade_tests.UpgradeTests.test_upgrade,items=5000,initial_version=5.0.0-3519,nodes_init=3,initialize_events=create_fts_index_query_compare,initial-services-setting=kv,fts-kv,fts-kv,fts,upgrade_services_in=same,after_events=rebalance_in-check_index_type-update_index_to_upside_down-run_fts_query_and_compare,after_upgrade_services_in=kv,fts,disable_HTP=True,upgrade_test=True,init_nodes=False,after_upgrade_index_type=upside_down,GROUP=ONLINE
Test Input params:
{'GROUP': 'ONLINE', 'after_upgrade_services_in': 'kv,fts', 'initial_version': '5.0.0-3519', 'upgrade_test': 'True', 'fts_quota': '512', 'init_nodes': 'False', 'after_events': 'rebalance_in-check_index_type-update_index_to_upside_down-run_fts_query_and_compare', 'upgrade_services_in': 'same', 'upgrade_version': '6.0.0-1564', 'case_number': 14, 'get-logs': 'False', 'initialize_events': 'create_fts_index_query_compare', 'nodes_init': '3', 'logs_folder': '/root/jenkins/workspace/centos-p0-fts-vset00-00-upgrade/logs/testrunner-18-Aug-26_14-55-28/test_14', 'after_upgrade_index_type': 'upside_down', 'items': '5000', 'conf_file': 'fts/py-fts-upgrade.conf', 'num_nodes': 4, 'cluster_name': 'testexec.14385', 'initial-services-setting': 'kv,fts-kv,fts-kv,fts', 'stop-on-failure': 'False', 'get-cbcollect-info': 'True', 'spec': 'py-fts-upgrade', 'disable_HTP': 'True', 'ini': '/tmp/testexec.14385.ini'}
[2018-08-26 17:06:32,538] - [remote_util:232] INFO - connecting to 172.23.97.230 with username:root password:couchbase ssh_key:
[2018-08-26 17:06:32,635] - [remote_util:266] INFO - Connected to 172.23.97.230
[2018-08-26 17:06:32,804] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:06:33,115] - [rest_client:2016] INFO - Node version in cluster 6.0.0-1564-enterprise
[2018-08-26 17:06:33,121] - [rest_client:811] ERROR - GET http://172.23.97.230:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password
[2018-08-26 17:06:33,122] - [rest_client:2026] INFO - Node versions in cluster []
[2018-08-26 17:06:33,122] - [basetestcase:206] INFO - ============== basetestcase setup was started for test #14 test_upgrade==============
[2018-08-26 17:06:33,128] - [rest_client:811] ERROR - GET http://172.23.97.230:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password
[2018-08-26 17:06:33,173] - [basetestcase:489] INFO - sleep for 10 secs. ...
[2018-08-26 17:06:43,182] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.97.230:8091
[2018-08-26 17:06:43,186] - [cluster_helper:80] INFO - ns_server @ 172.23.97.230:8091 is running
[2018-08-26 17:06:43,190] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.98.11:8091
[2018-08-26 17:06:43,194] - [cluster_helper:80] INFO - ns_server @ 172.23.98.11:8091 is running
[2018-08-26 17:06:43,199] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.98.19:8091
[2018-08-26 17:06:43,202] - [cluster_helper:80] INFO - ns_server @ 172.23.98.19:8091 is running
[2018-08-26 17:06:43,207] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.98.214:8091
[2018-08-26 17:06:43,211] - [cluster_helper:80] INFO - ns_server @ 172.23.98.214:8091 is running
[2018-08-26 17:06:43,211] - [basetestcase:251] INFO - initializing cluster
[2018-08-26 17:06:43,211] - [basetestcase:527] INFO - service will set to first element
[2018-08-26 17:06:43,212] - [basetestcase:527] INFO - service will set to first element
[2018-08-26 17:06:43,212] - [basetestcase:527] INFO - service will set to first element
[2018-08-26 17:06:43,212] - [basetestcase:527] INFO - service will set to first element
[2018-08-26 17:06:43,556] - [task:125] INFO - server: ip:172.23.97.230 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.97.230', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.97.230', 'uptime': u'156', 'mcdMemoryReserved': 3230, 'storageTotalRam': 4038, 'hostname': u'172.23.97.230:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f44cd40d090>], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-1564-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3434668032, 'memoryTotal': 4234354688, 'memoryQuota': 1757, 'mcdMemoryAllocated': 3230, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-08-26 17:06:43,556] - [rest_client:917] INFO - pools/default params : memoryQuota=2153
[2018-08-26 17:06:43,566] - [rest_client:1480] INFO - /diag/eval status on 172.23.97.230:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:06:43,568] - [rest_client:1480] INFO - /diag/eval status on 172.23.97.230:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:06:43,568] - [rest_client:952] INFO - settings/indexes params : storageMode=plasma
[2018-08-26 17:06:43,573] - [rest_client:895] INFO - /node/controller/setupServices params on 172.23.97.230: 8091:services=kv&password=password&hostname=172.23.97.230&user=Administrator
[2018-08-26 17:06:43,579] - [rest_client:835] INFO - settings/web params on 172.23.97.230:8091:username=Administrator&password=password&port=8091
[2018-08-26 17:06:43,616] - [task:125] INFO - server: ip:172.23.98.11 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.98.11', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.98.11', 'uptime': u'268', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.98.11:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f44cd40d090>], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-1564-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'fts', u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 2530222080, 'memoryTotal': 3961200640, 'memoryQuota': 2014, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-08-26 17:06:43,617] - [rest_client:917] INFO - pools/default params : memoryQuota=2014
[2018-08-26 17:06:43,624] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.11:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:06:43,627] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.11:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:06:43,627] - [rest_client:952] INFO - settings/indexes params : storageMode=plasma
[2018-08-26 17:06:43,635] - [rest_client:895] INFO - /node/controller/setupServices params on 172.23.98.11: 8091:services=kv&password=password&hostname=172.23.98.11&user=Administrator
[2018-08-26 17:06:43,639] - [rest_client:811] ERROR - POST http://172.23.98.11:8091//node/controller/setupServices body: services=kv&password=password&hostname=172.23.98.11&user=Administrator headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown ["cannot change node services after cluster is provisioned"] auth: Administrator:password
[2018-08-26 17:06:43,639] - [rest_client:900] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2018-08-26 17:06:43,639] - [rest_client:835] INFO - settings/web params on 172.23.98.11:8091:username=Administrator&password=password&port=8091
[2018-08-26 17:06:43,688] - [task:125] INFO - server: ip:172.23.98.19 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.98.19', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.98.19', 'uptime': u'376', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.98.19:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f44cd40d090>], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-1564-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3189895168, 'memoryTotal': 3961200640, 'memoryQuota': 1600, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-08-26 17:06:43,688] - [rest_client:917] INFO - pools/default params : memoryQuota=2014
[2018-08-26 17:06:43,697] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.19:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:06:43,699] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.19:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:06:43,699] - [rest_client:952] INFO - settings/indexes params : storageMode=plasma
[2018-08-26 17:06:43,705] - [rest_client:895] INFO - /node/controller/setupServices params on 172.23.98.19: 8091:services=kv&password=password&hostname=172.23.98.19&user=Administrator
[2018-08-26 17:06:43,709] - [rest_client:835] INFO - settings/web params on 172.23.98.19:8091:username=Administrator&password=password&port=8091
[2018-08-26 17:06:43,750] - [task:125] INFO - server: ip:172.23.98.214 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.98.214', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.98.214', 'uptime': u'498', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3778, 'hostname': u'172.23.98.214:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f44cd40d090>], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-1564-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3087523840, 'memoryTotal': 3961724928, 'memoryQuota': 1601, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-08-26 17:06:43,750] - [rest_client:917] INFO - pools/default params : memoryQuota=2014
[2018-08-26 17:06:43,760] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.214:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:06:43,763] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.214:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:06:43,763] - [rest_client:952] INFO - settings/indexes params : storageMode=plasma
[2018-08-26 17:06:43,769] - [rest_client:895] INFO - /node/controller/setupServices params on 172.23.98.214: 8091:services=kv&password=password&hostname=172.23.98.214&user=Administrator
[2018-08-26 17:06:43,773] - [rest_client:835] INFO - settings/web params on 172.23.98.214:8091:username=Administrator&password=password&port=8091
[2018-08-26 17:06:43,817] - [basetestcase:2091] INFO - **** add built-in 'cbadminbucket' user to node 172.23.97.230 ****
[2018-08-26 17:06:43,828] - [rest_client:811] ERROR - DELETE http://172.23.97.230:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password
[2018-08-26 17:06:43,828] - [internal_user:36] INFO - Exception while deleting user. Exception is -"User was not found."
[2018-08-26 17:06:43,851] - [basetestcase:489] INFO - sleep for 10 secs. ...
[2018-08-26 17:06:53,862] - [basetestcase:2096] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2018-08-26 17:06:53,875] - [basetestcase:489] INFO - sleep for 10 secs. ...
[2018-08-26 17:07:03,885] - [basetestcase:274] INFO - done initializing cluster
[2018-08-26 17:07:04,830] - [task:446] INFO - adding node 172.23.98.11:8091 to cluster
[2018-08-26 17:07:04,830] - [rest_client:1244] INFO - adding remote node @172.23.98.11:8091 to this cluster @172.23.97.230:8091
[2018-08-26 17:07:06,583] - [task:446] INFO - adding node 172.23.98.19:8091 to cluster
[2018-08-26 17:07:06,583] - [rest_client:1244] INFO - adding remote node @172.23.98.19:8091 to this cluster @172.23.97.230:8091
[2018-08-26 17:07:08,315] - [rest_client:1462] INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.98.19,ns_1@172.23.97.230,ns_1@172.23.98.11'}
[2018-08-26 17:07:08,326] - [rest_client:1467] INFO - rebalance operation started
[2018-08-26 17:07:08,329] - [rest_client:1615] INFO - rebalance percentage : 0.00 %
[2018-08-26 17:07:08,329] - [task:512] INFO - Rebalance - status: running, progress: 0.00%
[2018-08-26 17:07:18,351] - [task:512] INFO - Rebalance - status: none, progress: 100.00%
[2018-08-26 17:07:18,356] - [task:571] INFO - rebalancing was completed with progress: 100% in 10.0297191143 sec
[2018-08-26 17:07:18,363] - [basetestcase:351] INFO - ============== basetestcase setup was finished for test #14 test_upgrade ==============
[2018-08-26 17:07:18,369] - [basetestcase:489] INFO - sleep for 10 secs. ...
[2018-08-26 17:07:28,379] - [newupgradebasetest:38] INFO - ============== NewUpgradeBaseTest setup has started ==============
[2018-08-26 17:07:28,381] - [remote_util:232] INFO - connecting to 172.23.97.230 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:28,463] - [remote_util:266] INFO - Connected to 172.23.97.230
[2018-08-26 17:07:28,669] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:28,992] - [newupgradebasetest:139] INFO - ============== NewUpgradeBaseTest setup has completed ==============
[2018-08-26 17:07:28,992] - [newupgradebasetest:195] INFO - will install 5.0.0-3519 on ['172.23.97.230', '172.23.98.11', '172.23.98.19']
[2018-08-26 17:07:28,994] - [remote_util:232] INFO - connecting to 172.23.97.230 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:28,999] - [remote_util:232] INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:28,999] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:29,079] - [remote_util:266] INFO - Connected to 172.23.97.230
[2018-08-26 17:07:29,098] - [remote_util:266] INFO - Connected to 172.23.98.11
[2018-08-26 17:07:29,098] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:07:29,259] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:29,352] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:29,383] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:29,537] - [remote_util:2404] INFO - 172.23.97.230 *****In couchbase uninstall****
[2018-08-26 17:07:29,537] - [remote_util:2412] INFO - CentOS
[2018-08-26 17:07:29,646] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: killall -9 rpm
[2018-08-26 17:07:29,656] - [remote_util:2404] INFO - 172.23.98.19 *****In couchbase uninstall****
[2018-08-26 17:07:29,656] - [remote_util:2412] INFO - CentOS
[2018-08-26 17:07:29,668] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:29,669] - [remote_util:2934] ERROR - rpm: no process found
[2018-08-26 17:07:29,669] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: rm -f /var/lib/rpm/.rpm.lock
[2018-08-26 17:07:29,672] - [remote_util:2404] INFO - 172.23.98.11 *****In couchbase uninstall****
[2018-08-26 17:07:29,672] - [remote_util:2412] INFO - CentOS
[2018-08-26 17:07:29,742] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:29,769] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: killall -9 rpm
[2018-08-26 17:07:29,787] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: killall -9 rpm
[2018-08-26 17:07:29,798] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:29,798] - [remote_util:2934] ERROR - rpm: no process found
[2018-08-26 17:07:29,798] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: rm -f /var/lib/rpm/.rpm.lock
[2018-08-26 17:07:29,817] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:29,818] - [remote_util:2934] ERROR - rpm: no process found
[2018-08-26 17:07:29,818] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: rm -f /var/lib/rpm/.rpm.lock
[2018-08-26 17:07:29,847] - [remote_util:2692] INFO - running rpm -e to remove couchbase-server
[2018-08-26 17:07:29,847] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: rpm -e couchbase-server
[2018-08-26 17:07:29,848] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:29,893] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:29,929] - [remote_util:2692] INFO - running rpm -e to remove couchbase-server
[2018-08-26 17:07:29,930] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: rpm -e couchbase-server
[2018-08-26 17:07:30,004] - [remote_util:2692] INFO - running rpm -e to remove couchbase-server
[2018-08-26 17:07:30,004] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: rpm -e couchbase-server
[2018-08-26 17:07:32,973] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:32,974] - [remote_util:2934] ERROR -
**** Uninstalling on server: 172.23.97.230 ****
[2018-08-26 17:07:32,974] - [remote_util:2934] ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory
[2018-08-26 17:07:32,974] - [remote_util:2934] ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave
[2018-08-26 17:07:32,974] - [remote_util:2934] ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave
[2018-08-26 17:07:32,975] - [remote_util:2934] ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave
[2018-08-26 17:07:32,975] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: pkill -u couchbase
[2018-08-26 17:07:32,990] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:32,990] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: ps -ef | grep couchbase
[2018-08-26 17:07:33,034] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:33,035] - [remote_util:2934] ERROR -
**** Uninstalling on server: 172.23.98.11 ****
[2018-08-26 17:07:33,035] - [remote_util:2934] ERROR - warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave
[2018-08-26 17:07:33,035] - [remote_util:2934] ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory
[2018-08-26 17:07:33,035] - [remote_util:2934] ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave
[2018-08-26 17:07:33,035] - [remote_util:2934] ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave
[2018-08-26 17:07:33,035] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: pkill -u couchbase
[2018-08-26 17:07:33,051] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:33,052] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: ps -ef | grep couchbase
[2018-08-26 17:07:33,076] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:33,077] - [remote_util:2972] INFO - root 12058 11889 0 17:07 ? 00:00:00 bash -c ps -ef | grep couchbase
[2018-08-26 17:07:33,077] - [remote_util:2972] INFO - root 12064 12058 0 17:07 ? 00:00:00 grep couchbase
[2018-08-26 17:07:33,097] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:33,098] - [remote_util:2934] ERROR -
**** Uninstalling on server: 172.23.98.19 ****
[2018-08-26 17:07:33,098] - [remote_util:2934] ERROR - warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave
[2018-08-26 17:07:33,098] - [remote_util:2934] ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip: remove failed: No such file or directory
[2018-08-26 17:07:33,098] - [remote_util:2934] ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave
[2018-08-26 17:07:33,098] - [remote_util:2934] ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave
[2018-08-26 17:07:33,098] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: pkill -u couchbase
[2018-08-26 17:07:33,115] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:33,115] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: ps -ef | grep couchbase
[2018-08-26 17:07:33,138] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:33,139] - [remote_util:2972] INFO - root 31493 31287 0 17:07 ? 00:00:00 bash -c ps -ef | grep couchbase
[2018-08-26 17:07:33,139] - [remote_util:2972] INFO - root 31501 31493 0 17:07 ? 00:00:00 grep couchbase
[2018-08-26 17:07:33,198] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:33,199] - [remote_util:2972] INFO - root 7579 7410 0 17:07 ? 00:00:00 bash -c ps -ef | grep couchbase
[2018-08-26 17:07:33,199] - [remote_util:2972] INFO - root 7585 7579 0 17:07 ? 00:00:00 grep couchbase
[2018-08-26 17:07:34,551] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: kill -9 $(ps aux | pgrep 'memcached')
[2018-08-26 17:07:34,574] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: kill -9 $(ps aux | pgrep 'memcached')
[2018-08-26 17:07:34,607] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:34,608] - [remote_util:2934] ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
[2018-08-26 17:07:34,608] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: ipcrm
[2018-08-26 17:07:34,633] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: kill -9 $(ps aux | pgrep 'memcached')
[2018-08-26 17:07:34,656] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:34,657] - [remote_util:2934] ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
[2018-08-26 17:07:34,657] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: ipcrm
[2018-08-26 17:07:34,681] - [remote_util:3174] INFO - command executed successfully
thread uninstaller-thread-172.23.97.230 finished
[2018-08-26 17:07:34,690] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:34,691] - [remote_util:2934] ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
[2018-08-26 17:07:34,691] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: ipcrm
[2018-08-26 17:07:34,731] - [remote_util:3174] INFO - command executed successfully
thread uninstaller-thread-172.23.98.11 finished
[2018-08-26 17:07:34,764] - [remote_util:3174] INFO - command executed successfully
thread uninstaller-thread-172.23.98.19 finished
[2018-08-26 17:07:34,766] - [remote_util:232] INFO - connecting to 172.23.97.230 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:34,863] - [remote_util:266] INFO - Connected to 172.23.97.230
[2018-08-26 17:07:35,033] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:35,431] - [remote_util:232] INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:35,527] - [remote_util:266] INFO - Connected to 172.23.98.11
[2018-08-26 17:07:35,780] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:36,216] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:36,312] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:07:36,585] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:36,930] - [install:629] INFO - ********CouchbaseServerInstaller:install
[2018-08-26 17:07:36,932] - [remote_util:232] INFO - connecting to 172.23.97.230 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:36,932] - [install:629] INFO - ********CouchbaseServerInstaller:install
[2018-08-26 17:07:36,933] - [install:629] INFO - ********CouchbaseServerInstaller:install
[2018-08-26 17:07:36,936] - [remote_util:232] INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:36,937] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:37,014] - [remote_util:266] INFO - Connected to 172.23.97.230
[2018-08-26 17:07:37,018] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:07:37,034] - [remote_util:266] INFO - Connected to 172.23.98.11
[2018-08-26 17:07:37,195] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:37,267] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:37,294] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
*** OS version of this server 172.23.97.230 is CentOS 7 ***
[2018-08-26 17:07:37,485] - [remote_util:232] INFO - connecting to 172.23.97.230 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:37,566] - [remote_util:266] INFO - Connected to 172.23.97.230
*** OS version of this server 172.23.98.19 is CentOS 7 ***
*** OS version of this server 172.23.98.11 is CentOS 7 ***
[2018-08-26 17:07:37,643] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:37,673] - [remote_util:232] INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:37,723] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:07:37,749] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:37,754] - [remote_util:266] INFO - Connected to 172.23.98.11
[2018-08-26 17:07:37,969] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:38,026] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:38,057] - [remote_util:792] INFO - Check if url http://172.23.120.24/builds/releases/5.0.0/couchbase-server-enterprise-5.0.0-centos7.x86_64.rpm is ok
[2018-08-26 17:07:38,126] - [remote_util:795] INFO - This url http://172.23.120.24/builds/releases/5.0.0/couchbase-server-enterprise-5.0.0-centos7.x86_64.rpm is live
[2018-08-26 17:07:38,127] - [remote_util:232] INFO - connecting to 172.23.97.230 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:38,207] - [remote_util:266] INFO - Connected to 172.23.97.230
[2018-08-26 17:07:38,273] - [remote_util:792] INFO - Check if url http://172.23.120.24/builds/releases/5.0.0/couchbase-server-enterprise-5.0.0-centos7.x86_64.rpm is ok
[2018-08-26 17:07:38,276] - [remote_util:795] INFO - This url http://172.23.120.24/builds/releases/5.0.0/couchbase-server-enterprise-5.0.0-centos7.x86_64.rpm is live
[2018-08-26 17:07:38,280] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:38,343] - [remote_util:792] INFO - Check if url http://172.23.120.24/builds/releases/5.0.0/couchbase-server-enterprise-5.0.0-centos7.x86_64.rpm is ok
[2018-08-26 17:07:38,345] - [remote_util:795] INFO - This url http://172.23.120.24/builds/releases/5.0.0/couchbase-server-enterprise-5.0.0-centos7.x86_64.rpm is live
[2018-08-26 17:07:38,346] - [remote_util:232] INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key:
[2018-08-26 17:07:38,379] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:07:38,415] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:38,445] - [remote_util:266] INFO - Connected to 172.23.98.11
[2018-08-26 17:07:38,600] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:38,691] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: /sbin/iptables -F
[2018-08-26 17:07:38,701] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:07:38,765] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:38,766] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: /sbin/iptables -t nat -F
[2018-08-26 17:07:38,840] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:38,841] - [remote_util:294] INFO - Connect to node: 172.23.97.230 as user: root
[2018-08-26 17:07:38,886] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: /sbin/iptables -F
[2018-08-26 17:07:38,922] - [remote_util:317] INFO - Connected to 172.23.97.230 as root
[2018-08-26 17:07:38,922] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_5.0.0-3519.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D
[2018-08-26 17:07:38,962] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:38,962] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: /sbin/iptables -t nat -F
[2018-08-26 17:07:38,992] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: /sbin/iptables -F
[2018-08-26 17:07:39,036] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:39,036] - [remote_util:294] INFO - Connect to node: 172.23.98.19 as user: root
[2018-08-26 17:07:39,068] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:39,069] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: /sbin/iptables -t nat -F
[2018-08-26 17:07:39,084] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:39,085] - [remote_util:2934] ERROR - mv: cannot stat ‘couchbase-server-enterprise_x86_64_5.0.0-3519.rpm’: No such file or directory
[2018-08-26 17:07:39,085] - [remote_util:2934] ERROR - mv: cannot stat ‘core.*’: No such file or directory
[2018-08-26 17:07:39,086] - [remote_util:2934] ERROR - rm: cannot remove ‘cb_hQxi’: Is a directory
[2018-08-26 17:07:39,086] - [remote_util:2934] ERROR - rm: cannot remove ‘export172.23.97.230’: Is a directory
[2018-08-26 17:07:39,086] - [remote_util:2934] ERROR - rm: cannot remove ‘systemd-private-0c527dbf05974f74a647fc5dce5902df-ntpd.service-F4vC1o’: Is a directory
[2018-08-26 17:07:39,086] - [remote_util:2934] ERROR - mv: cannot stat ‘cb_hQxi/*’: No such file or directory
[2018-08-26 17:07:39,086] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: cd /tmp;wget -q -O couchbase-server-enterprise_x86_64_5.0.0-3519.rpm http://172.23.120.24/builds/releases/5.0.0/couchbase-server-enterprise-5.0.0-centos7.x86_64.rpm;cd /tmp;ls -lh
[2018-08-26 17:07:39,134] - [remote_util:317] INFO - Connected to 172.23.98.19 as root
[2018-08-26 17:07:39,134] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_5.0.0-3519.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D
[2018-08-26 17:07:39,143] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:39,144] - [remote_util:294] INFO - Connect to node: 172.23.98.11 as user: root
[2018-08-26 17:07:39,241] - [remote_util:317] INFO - Connected to 172.23.98.11 as root
[2018-08-26 17:07:39,241] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_5.0.0-3519.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D
[2018-08-26 17:07:39,362] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:39,362] - [remote_util:2934] ERROR - mv: cannot stat ‘couchbase-server-enterprise_x86_64_5.0.0-3519.rpm’: No such file or directory
[2018-08-26 17:07:39,362] - [remote_util:2934] ERROR - mv: cannot stat ‘core.*’: No such file or directory
[2018-08-26 17:07:39,363] - [remote_util:2934] ERROR - rm: cannot remove ‘cb_sk02’: Is a directory
[2018-08-26 17:07:39,363] - [remote_util:2934] ERROR - rm: cannot remove ‘entbackup’: Is a directory
[2018-08-26 17:07:39,363] - [remote_util:2934] ERROR - rm: cannot remove ‘systemd-private-8dced1e9577f43b3a692103e0615fe24-ntpd.service-Ah1cAQ’: Is a directory
[2018-08-26 17:07:39,363] - [remote_util:2934] ERROR - mv: cannot stat ‘cb_sk02/*’: No such file or directory
[2018-08-26 17:07:39,363] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: cd /tmp;wget -q -O couchbase-server-enterprise_x86_64_5.0.0-3519.rpm http://172.23.120.24/builds/releases/5.0.0/couchbase-server-enterprise-5.0.0-centos7.x86_64.rpm;cd /tmp;ls -lh
[2018-08-26 17:07:39,471] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:39,471] - [remote_util:2934] ERROR - mv: cannot stat ‘couchbase-server-enterprise_x86_64_5.0.0-3519.rpm’: No such file or directory
[2018-08-26 17:07:39,472] - [remote_util:2934] ERROR - mv: cannot stat ‘core.*’: No such file or directory
[2018-08-26 17:07:39,472] - [remote_util:2934] ERROR - rm: cannot remove ‘cb_0FyP’: Is a directory
[2018-08-26 17:07:39,472] - [remote_util:2934] ERROR - rm: cannot remove ‘hsperfdata_couchbase’: Is a directory
[2018-08-26 17:07:39,472] - [remote_util:2934] ERROR - rm: cannot remove ‘systemd-private-a2560737ebc14d3685e109fc852c1908-ntpd.service-ZWT5GC’: Is a directory
[2018-08-26 17:07:39,472] - [remote_util:2934] ERROR - mv: cannot stat ‘cb_0FyP/*’: No such file or directory
[2018-08-26 17:07:39,472] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: cd /tmp;wget -q -O couchbase-server-enterprise_x86_64_5.0.0-3519.rpm http://172.23.120.24/builds/releases/5.0.0/couchbase-server-enterprise-5.0.0-centos7.x86_64.rpm;cd /tmp;ls -lh
[2018-08-26 17:07:40,198] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:40,199] - [remote_util:2972] INFO - total 115M
[2018-08-26 17:07:40,199] - [remote_util:2972] INFO - -rw-r--r--. 1 root root 115M Sep 14 2017 couchbase-server-enterprise_x86_64_5.0.0-3519.rpm
[2018-08-26 17:07:40,199] - [remote_util:2972] INFO - drwxr-xr-x. 2 root root 20 Aug 25 09:06 export172.23.97.230
[2018-08-26 17:07:40,199] - [remote_util:2972] INFO - drwx------. 3 root root 16 Aug 17 15:26 systemd-private-0c527dbf05974f74a647fc5dce5902df-ntpd.service-F4vC1o
[2018-08-26 17:07:40,279] - [remote_util:1832] INFO - *****install server ***
[2018-08-26 17:07:40,279] - [remote_util:1847] INFO - deliverable_type : rpm
[2018-08-26 17:07:40,279] - [remote_util:1888] INFO - /tmp/couchbase-server-enterprise_x86_64_5.0.0-3519.rpm or /tmp/couchbase-server-enterprise
[2018-08-26 17:07:40,279] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: /sbin/sysctl vm.swappiness=0
[2018-08-26 17:07:40,353] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:40,353] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: cat /etc/redhat-release
[2018-08-26 17:07:40,427] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:40,428] - [remote_util:2972] INFO - CentOS Linux release 7.2.1511 (Core)
[2018-08-26 17:07:40,428] - [remote_util:4480] INFO - no need to install pkgconfig
[2018-08-26 17:07:41,216] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:41,217] - [remote_util:2972] INFO - total 115M
[2018-08-26 17:07:41,217] - [remote_util:2972] INFO - -rw-r--r--. 1 root root 115M Sep 14 2017 couchbase-server-enterprise_x86_64_5.0.0-3519.rpm
[2018-08-26 17:07:41,217] - [remote_util:2972] INFO - drwxrwx---. 4 root root 44 Aug 25 09:00 entbackup
[2018-08-26 17:07:41,217] - [remote_util:2972] INFO - drwx------. 3 root root 16 Aug 26 06:27 systemd-private-8dced1e9577f43b3a692103e0615fe24-ntpd.service-Ah1cAQ
[2018-08-26 17:07:41,296] - [remote_util:1832] INFO - *****install server ***
[2018-08-26 17:07:41,296] - [remote_util:1847] INFO - deliverable_type : rpm
[2018-08-26 17:07:41,296] - [remote_util:1888] INFO - /tmp/couchbase-server-enterprise_x86_64_5.0.0-3519.rpm or /tmp/couchbase-server-enterprise
[2018-08-26 17:07:41,296] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: /sbin/sysctl vm.swappiness=0
[2018-08-26 17:07:41,372] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:41,372] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: cat /etc/redhat-release
[2018-08-26 17:07:41,445] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:41,445] - [remote_util:2972] INFO - CentOS Linux release 7.3.1611 (Core)
[2018-08-26 17:07:41,445] - [remote_util:4480] INFO - no need to install pkgconfig
[2018-08-26 17:07:42,237] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:42,237] - [remote_util:2972] INFO - total 115M
[2018-08-26 17:07:42,237] - [remote_util:2972] INFO - -rw-r--r--. 1 root root 115M Sep 14 2017 couchbase-server-enterprise_x86_64_5.0.0-3519.rpm
[2018-08-26 17:07:42,237] - [remote_util:2972] INFO - drwxr-x---. 2 couchbase couchbase 6 Aug 16 18:54 hsperfdata_couchbase
[2018-08-26 17:07:42,237] - [remote_util:2972] INFO - drwx------. 3 root root 16 Aug 7 13:22 systemd-private-a2560737ebc14d3685e109fc852c1908-ntpd.service-ZWT5GC
[2018-08-26 17:07:42,318] - [remote_util:1832] INFO - *****install server ***
[2018-08-26 17:07:42,318] - [remote_util:1847] INFO - deliverable_type : rpm
[2018-08-26 17:07:42,318] - [remote_util:1888] INFO - /tmp/couchbase-server-enterprise_x86_64_5.0.0-3519.rpm or /tmp/couchbase-server-enterprise
[2018-08-26 17:07:42,318] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: /sbin/sysctl vm.swappiness=0
[2018-08-26 17:07:42,394] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:42,394] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: cat /etc/redhat-release
[2018-08-26 17:07:42,468] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:42,469] - [remote_util:2972] INFO - CentOS Linux release 7.3.1611 (Core)
[2018-08-26 17:07:42,469] - [remote_util:4480] INFO - no need to install pkgconfig
[2018-08-26 17:07:54,191] - [remote_util:1679] ERROR - Directory at /tmp DOES exist. Fx returns True
[2018-08-26 17:07:54,279] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server
[2018-08-26 17:07:54,353] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:54,455] - [remote_util:3582] INFO - Running systemd command on this server
[2018-08-26 17:07:54,456] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: systemctl start couchbase-server.service
[2018-08-26 17:07:54,491] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:54,563] - [install:731] INFO - wait 5 seconds for Couchbase server to start
[2018-08-26 17:07:55,509] - [remote_util:1679] ERROR - Directory at /tmp DOES exist. Fx returns True
[2018-08-26 17:07:55,577] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server
[2018-08-26 17:07:55,627] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:55,731] - [remote_util:3582] INFO - Running systemd command on this server
[2018-08-26 17:07:55,731] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: systemctl start couchbase-server.service
[2018-08-26 17:07:55,767] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:55,840] - [install:731] INFO - wait 5 seconds for Couchbase server to start
[2018-08-26 17:07:56,503] - [remote_util:1679] ERROR - Directory at /tmp DOES exist. Fx returns True
[2018-08-26 17:07:56,595] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server
[2018-08-26 17:07:56,671] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:56,781] - [remote_util:3582] INFO - Running systemd command on this server
[2018-08-26 17:07:56,782] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: systemctl start couchbase-server.service
[2018-08-26 17:07:56,818] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:07:56,870] - [install:731] INFO - wait 5 seconds for Couchbase server to start
thread installer-thread-172.23.97.230 finished
thread installer-thread-172.23.98.11 finished
thread installer-thread-172.23.98.19 finished
[2018-08-26 17:08:01,887] - [remote_util:232] INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key:
[2018-08-26 17:08:01,888] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:08:01,889] - [remote_util:232] INFO - connecting to 172.23.97.230 with username:root password:couchbase ssh_key:
[2018-08-26 17:08:01,989] - [remote_util:266] INFO - Connected to 172.23.98.11
[2018-08-26 17:08:01,989] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:08:01,989] - [remote_util:266] INFO - Connected to 172.23.97.230
[2018-08-26 17:08:02,136] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:08:02,248] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:08:02,250] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:08:02,491] - [remote_util:749] INFO - 172.23.97.230 **** The version file /opt/couchbase/ VERSION.txt exists
[2018-08-26 17:08:02,674] - [remote_util:749] INFO - 172.23.98.11 **** The version file /opt/couchbase/ VERSION.txt exists
[2018-08-26 17:08:02,684] - [remote_util:749] INFO - 172.23.98.19 **** The version file /opt/couchbase/ VERSION.txt exists
[2018-08-26 17:08:05,514] - [rest_client:2016] INFO - Node version in cluster 5.0.0-3519-enterprise
[2018-08-26 17:08:05,515] - [remote_util:4597] INFO - ***** set NS_SERVER_CBAUTH env in linux *****
[2018-08-26 17:08:05,699] - [rest_client:2016] INFO - Node version in cluster 5.0.0-3519-enterprise
[2018-08-26 17:08:05,702] - [remote_util:4597] INFO - ***** set NS_SERVER_CBAUTH env in linux *****
[2018-08-26 17:08:05,711] - [rest_client:2016] INFO - Node version in cluster 5.0.0-3519-enterprise
[2018-08-26 17:08:05,711] - [remote_util:4597] INFO - ***** set NS_SERVER_CBAUTH env in linux *****
[2018-08-26 17:08:05,746] - [remote_util:4483] INFO - check if man installed on vm?
[2018-08-26 17:08:05,746] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: rpm -qa | grep 'man-db'
[2018-08-26 17:08:05,915] - [remote_util:4483] INFO - check if man installed on vm?
[2018-08-26 17:08:05,915] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: rpm -qa | grep 'man-db'
[2018-08-26 17:08:06,010] - [remote_util:4483] INFO - check if man installed on vm?
[2018-08-26 17:08:06,011] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: rpm -qa | grep 'man-db'
[2018-08-26 17:08:06,208] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:08:06,209] - [remote_util:4344] INFO -
---> Run command unzip to check if it is ready on server 172.23.97.230
[2018-08-26 17:08:06,209] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: unzip
[2018-08-26 17:08:06,282] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:08:06,282] - [remote_util:4376] INFO - unzip command is ready
[2018-08-26 17:08:06,283] - [remote_util:810] INFO - Check if ntp is installed
[2018-08-26 17:08:06,283] - [remote_util:813] INFO -
This OS version centos 7
[2018-08-26 17:08:06,283] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: systemctl status ntpd
[2018-08-26 17:08:06,368] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:08:06,369] - [remote_util:3137] INFO - running command.raw on 172.23.97.230: date
[2018-08-26 17:08:06,442] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:08:06,541] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:08:06,542] - [remote_util:4344] INFO -
---> Run command unzip to check if it is ready on server 172.23.98.19
[2018-08-26 17:08:06,542] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: unzip
[2018-08-26 17:08:06,556] - [remote_util:882] INFO -
['Sun Aug 26 17:08:06 PDT 2018'] IP: 172.23.97.230
thread initializer-thread-172.23.97.230 finished
[2018-08-26 17:08:06,616] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:08:06,616] - [remote_util:4376] INFO - unzip command is ready
[2018-08-26 17:08:06,616] - [remote_util:810] INFO - Check if ntp is installed
[2018-08-26 17:08:06,617] - [remote_util:813] INFO -
This OS version centos 7
[2018-08-26 17:08:06,617] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: systemctl status ntpd
[2018-08-26 17:08:06,637] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:08:06,637] - [remote_util:4344] INFO -
---> Run command unzip to check if it is ready on server 172.23.98.11
[2018-08-26 17:08:06,637] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: unzip
[2018-08-26 17:08:06,700] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:08:06,700] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: date
[2018-08-26 17:08:06,712] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:08:06,712] - [remote_util:4376] INFO - unzip command is ready
[2018-08-26 17:08:06,712] - [remote_util:810] INFO - Check if ntp is installed
[2018-08-26 17:08:06,713] - [remote_util:813] INFO -
This OS version centos 7
[2018-08-26 17:08:06,713] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: systemctl status ntpd
[2018-08-26 17:08:06,772] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:08:06,796] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:08:06,796] - [remote_util:3137] INFO - running command.raw on 172.23.98.11: date
[2018-08-26 17:08:06,847] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:08:06,889] - [remote_util:882] INFO -
['Sun Aug 26 17:08:06 PDT 2018'] IP: 172.23.98.19
[2018-08-26 17:08:06,990] - [remote_util:882] INFO -
['Sun Aug 26 17:08:06 PDT 2018'] IP: 172.23.98.11
thread initializer-thread-172.23.98.11 finished
thread initializer-thread-172.23.98.19 finished
[2018-08-26 17:08:06,993] - [remote_util:232] INFO - connecting to 172.23.97.230 with username:root password:couchbase ssh_key:
[2018-08-26 17:08:07,089] - [remote_util:266] INFO - Connected to 172.23.97.230
[2018-08-26 17:08:07,236] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:08:07,539] - [remote_util:232] INFO - connecting to 172.23.97.230 with username:root password:couchbase ssh_key:
[2018-08-26 17:08:07,619] - [remote_util:266] INFO - Connected to 172.23.97.230
[2018-08-26 17:08:07,826] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:08:08,229] - [remote_util:749] INFO - 172.23.97.230 **** The version file /opt/couchbase/ VERSION.txt exists
[2018-08-26 17:08:08,229] - [basetestcase:489] INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ...
[2018-08-26 17:08:13,361] - [remote_util:232] INFO - connecting to 172.23.98.11 with username:root password:couchbase ssh_key:
[2018-08-26 17:08:13,458] - [remote_util:266] INFO - Connected to 172.23.98.11
[2018-08-26 17:08:13,731] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:08:14,056] - [remote_util:749] INFO - 172.23.98.11 **** The version file /opt/couchbase/ VERSION.txt exists
[2018-08-26 17:08:14,057] - [basetestcase:489] INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ...
[2018-08-26 17:08:19,184] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:08:19,281] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:08:19,528] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:08:19,935] - [remote_util:749] INFO - 172.23.98.19 **** The version file /opt/couchbase/ VERSION.txt exists
[2018-08-26 17:08:19,935] - [basetestcase:489] INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ...
[2018-08-26 17:08:25,432] - [task:125] INFO - server: ip:172.23.97.230 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'27', 'mcdMemoryReserved': 3230, 'storageTotalRam': 4038, 'hostname': u'172.23.97.230:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f44d5b42a50>], 'moxi': 11211, 'port': u'8091', 'version': u'5.0.0-3519-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327680, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3686957056, 'memoryTotal': 4234354688, 'memoryQuota': 2422, 'mcdMemoryAllocated': 3230, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-08-26 17:08:25,433] - [rest_client:917] INFO - pools/default params : memoryQuota=2153
[2018-08-26 17:08:25,439] - [rest_client:1480] INFO - /diag/eval status on 172.23.97.230:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:08:25,442] - [rest_client:1480] INFO - /diag/eval status on 172.23.97.230:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:08:25,442] - [rest_client:952] INFO - settings/indexes params : storageMode=forestdb
[2018-08-26 17:08:25,447] - [rest_client:811] ERROR - POST http://172.23.97.230:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in Spock enterprise edition."}} auth: Administrator:password
[2018-08-26 17:08:25,448] - [rest_client:895] INFO - /node/controller/setupServices params on 172.23.97.230: 8091:services=kv%2Cfts&password=password&hostname=172.23.97.230&user=Administrator
[2018-08-26 17:08:25,451] - [rest_client:835] INFO - settings/web params on 172.23.97.230:8091:username=Administrator&password=password&port=8091
[2018-08-26 17:08:25,514] - [task:125] INFO - server: ip:172.23.98.11 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'23', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.98.11:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f43f2914750>], 'moxi': 11211, 'port': u'8091', 'version': u'5.0.0-3519-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327680, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3379961856, 'memoryTotal': 3961200640, 'memoryQuota': 2266, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-08-26 17:08:25,514] - [rest_client:917] INFO - pools/default params : memoryQuota=2014
[2018-08-26 17:08:25,519] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.11:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:08:25,522] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.11:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:08:25,522] - [rest_client:952] INFO - settings/indexes params : storageMode=forestdb
[2018-08-26 17:08:25,527] - [rest_client:811] ERROR - POST http://172.23.98.11:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in Spock enterprise edition."}} auth: Administrator:password
[2018-08-26 17:08:25,528] - [rest_client:895] INFO - /node/controller/setupServices params on 172.23.98.11: 8091:services=kv%2Cfts&password=password&hostname=172.23.98.11&user=Administrator
[2018-08-26 17:08:25,531] - [rest_client:835] INFO - settings/web params on 172.23.98.11:8091:username=Administrator&password=password&port=8091
[2018-08-26 17:08:25,589] - [task:125] INFO - server: ip:172.23.98.19 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'23', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.98.19:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f44d5b42f50>], 'moxi': 11211, 'port': u'8091', 'version': u'5.0.0-3519-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327680, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3438882816, 'memoryTotal': 3961200640, 'memoryQuota': 2266, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-08-26 17:08:25,589] - [rest_client:917] INFO - pools/default params : memoryQuota=2014
[2018-08-26 17:08:25,595] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.19:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:08:25,597] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.19:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:08:25,597] - [rest_client:952] INFO - settings/indexes params : storageMode=forestdb
[2018-08-26 17:08:25,602] - [rest_client:811] ERROR - POST http://172.23.98.19:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in Spock enterprise edition."}} auth: Administrator:password
[2018-08-26 17:08:25,602] - [rest_client:895] INFO - /node/controller/setupServices params on 172.23.98.19: 8091:services=kv%2Cfts&password=password&hostname=172.23.98.19&user=Administrator
[2018-08-26 17:08:25,605] - [rest_client:835] INFO - settings/web params on 172.23.98.19:8091:username=Administrator&password=password&port=8091
[2018-08-26 17:08:26,653] - [task:446] INFO - adding node 172.23.98.11:8091 to cluster
[2018-08-26 17:08:26,653] - [rest_client:1244] INFO - adding remote node @172.23.98.11:8091 to this cluster @172.23.97.230:8091
[2018-08-26 17:08:28,750] - [rest_client:1462] INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.97.230,ns_1@172.23.98.11'}
[2018-08-26 17:08:28,757] - [rest_client:1467] INFO - rebalance operation started
[2018-08-26 17:08:28,759] - [rest_client:1615] INFO - rebalance percentage : 0.00 %
[2018-08-26 17:08:28,760] - [task:512] INFO - Rebalance - status: running, progress: 0.00%
[2018-08-26 17:08:38,779] - [task:512] INFO - Rebalance - status: none, progress: 100.00%
[2018-08-26 17:08:38,783] - [task:571] INFO - rebalancing was completed with progress: 100% in 10.0263590813 sec
[2018-08-26 17:08:38,783] - [basetestcase:489] INFO - sleep for 10 secs. ...
[2018-08-26 17:08:49,795] - [task:446] INFO - adding node 172.23.98.19:8091 to cluster
[2018-08-26 17:08:49,796] - [rest_client:1244] INFO - adding remote node @172.23.98.19:8091 to this cluster @172.23.97.230:8091
[2018-08-26 17:08:51,980] - [rest_client:1462] INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.98.19,ns_1@172.23.97.230,ns_1@172.23.98.11'}
[2018-08-26 17:08:51,985] - [rest_client:1467] INFO - rebalance operation started
[2018-08-26 17:08:51,988] - [rest_client:1615] INFO - rebalance percentage : 0.00 %
[2018-08-26 17:08:51,988] - [task:512] INFO - Rebalance - status: running, progress: 0.00%
[2018-08-26 17:09:02,007] - [task:512] INFO - Rebalance - status: none, progress: 100.00%
[2018-08-26 17:09:02,012] - [task:571] INFO - rebalancing was completed with progress: 100% in 10.0272479057 sec
[2018-08-26 17:09:02,013] - [basetestcase:489] INFO - sleep for 10 secs. ...
[2018-08-26 17:09:12,023] - [basetestcase:489] INFO - sleep for 10 secs. ...
[2018-08-26 17:09:22,043] - [task:125] INFO - server: ip:172.23.97.230 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.97.230', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.97.230', 'uptime': u'82', 'mcdMemoryReserved': 3230, 'storageTotalRam': 4038, 'hostname': u'172.23.97.230:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f43f2914650>], 'moxi': 11211, 'port': u'8091', 'version': u'5.0.0-3519-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327680, 'curr_items': 0, 'services': [u'fts', u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3663585280, 'memoryTotal': 4234354688, 'memoryQuota': 2153, 'mcdMemoryAllocated': 3230, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-08-26 17:09:22,043] - [rest_client:917] INFO - pools/default params : memoryQuota=2153
[2018-08-26 17:09:22,049] - [rest_client:1480] INFO - /diag/eval status on 172.23.97.230:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:09:22,052] - [rest_client:1480] INFO - /diag/eval status on 172.23.97.230:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:09:22,052] - [rest_client:952] INFO - settings/indexes params : storageMode=forestdb
[2018-08-26 17:09:22,055] - [rest_client:811] ERROR - POST http://172.23.97.230:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in Spock enterprise edition."}} auth: Administrator:password
[2018-08-26 17:09:22,055] - [rest_client:835] INFO - settings/web params on 172.23.97.230:8091:username=Administrator&password=password&port=8091
[2018-08-26 17:09:22,098] - [task:125] INFO - server: ip:172.23.98.11 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.98.11', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.98.11', 'uptime': u'80', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.98.11:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f44d5b42e90>], 'moxi': 11211, 'port': u'8091', 'version': u'5.0.0-3519-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327680, 'curr_items': 0, 'services': [u'fts', u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3362107392, 'memoryTotal': 3961200640, 'memoryQuota': 2153, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-08-26 17:09:22,098] - [rest_client:917] INFO - pools/default params : memoryQuota=2014
[2018-08-26 17:09:22,105] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.11:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:09:22,107] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.11:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:09:22,107] - [rest_client:952] INFO - settings/indexes params : storageMode=forestdb
[2018-08-26 17:09:22,110] - [rest_client:811] ERROR - POST http://172.23.98.11:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in Spock enterprise edition."}} auth: Administrator:password
[2018-08-26 17:09:22,110] - [rest_client:835] INFO - settings/web params on 172.23.98.11:8091:username=Administrator&password=password&port=8091
[2018-08-26 17:09:22,153] - [task:125] INFO - server: ip:172.23.98.19 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.98.19', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.98.19', 'uptime': u'84', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.98.19:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f44dc287f90>], 'moxi': 11211, 'port': u'8091', 'version': u'5.0.0-3519-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327680, 'curr_items': 0, 'services': [u'fts', u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3425177600, 'memoryTotal': 3961200640, 'memoryQuota': 2014, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2018-08-26 17:09:22,153] - [rest_client:917] INFO - pools/default params : memoryQuota=2014
[2018-08-26 17:09:22,159] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.19:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:09:22,162] - [rest_client:1480] INFO - /diag/eval status on 172.23.98.19:8091: True content: [5,0] command: cluster_compat_mode:get_compat_version().
[2018-08-26 17:09:22,162] - [rest_client:952] INFO - settings/indexes params : storageMode=forestdb
[2018-08-26 17:09:22,165] - [rest_client:811] ERROR - POST http://172.23.98.19:8091/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown {"errors":{"storageMode":"Storage mode cannot be set to 'forestdb' in Spock enterprise edition."}} auth: Administrator:password
[2018-08-26 17:09:22,165] - [rest_client:835] INFO - settings/web params on 172.23.98.19:8091:username=Administrator&password=password&port=8091
[2018-08-26 17:09:22,208] - [basetestcase:2091] INFO - **** add built-in 'cbadminbucket' user to node 172.23.97.230 ****
[2018-08-26 17:09:22,221] - [rest_client:811] ERROR - DELETE http://172.23.97.230:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password
[2018-08-26 17:09:22,221] - [internal_user:36] INFO - Exception while deleting user. Exception is -"User was not found."
[2018-08-26 17:09:22,253] - [basetestcase:489] INFO - sleep for 10 secs. ...
[2018-08-26 17:09:32,263] - [basetestcase:2096] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2018-08-26 17:09:32,277] - [basetestcase:489] INFO - sleep for 10 secs. ...
[2018-08-26 17:09:42,287] - [upgrade_tests:484] INFO - create_buckets
[2018-08-26 17:09:43,240] - [rest_client:2307] INFO - http://172.23.97.230:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=2014&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1
[2018-08-26 17:09:43,245] - [rest_client:2332] INFO - 0.01 seconds to create bucket default
[2018-08-26 17:09:43,246] - [bucket_helper:344] INFO - waiting for memcached bucket : default in 172.23.97.230 to accept set ops
[2018-08-26 17:09:44,365] - [data_helper:295] INFO - creating direct client 172.23.97.230:11210 default
[2018-08-26 17:09:44,419] - [data_helper:295] INFO - creating direct client 172.23.98.11:11210 default
[2018-08-26 17:09:44,478] - [data_helper:295] INFO - creating direct client 172.23.98.19:11210 default
[2018-08-26 17:09:44,560] - [data_helper:295] INFO - creating direct client 172.23.97.230:11210 default
[2018-08-26 17:09:44,615] - [data_helper:295] INFO - creating direct client 172.23.98.11:11210 default
[2018-08-26 17:09:44,681] - [data_helper:295] INFO - creating direct client 172.23.98.19:11210 default
[2018-08-26 17:09:44,970] - [task:332] INFO - bucket 'default' was created with per node RAM quota: 2014
[2018-08-26 17:09:45,017] - [basetestcase:489] INFO - sleep for 5 secs. sleep after create bucket ...
[2018-08-26 17:09:50,034] - [upgrade_tests:140] INFO -
*** Start init operations before upgrade begins ***
[2018-08-26 17:09:50,048] - [rest_client:2690] INFO - SUCCESS: FTS RAM quota set to 512mb
[2018-08-26 17:09:50,102] - [es_base:214] ERROR - http://172.23.96.146:9200/es_index error 404 reason: RemoteTransportException[[Digitek][inet[/172.23.99.253:9300]][indices:admin/delete]]; nested: IndexMissingException[[es_index] missing]; {"error":"RemoteTransportException[[Digitek][inet[/172.23.99.253:9300]][indices:admin/delete]]; nested: IndexMissingException[[es_index] missing]; ","status":404}
[2018-08-26 17:09:50,147] - [fts_callable:57] INFO - Created empty index 'es_index' on Elastic Search node with custom standard analyzer(default)
[2018-08-26 17:09:50,178] - [fts_base:895] INFO - Checking if index already exists ...
[2018-08-26 17:09:50,179] - [rest_client:816] ERROR - socket error while connecting to http://172.23.98.11:8094/api/index/index_default error [Errno 111] Connection refused
[2018-08-26 17:09:53,185] - [rest_client:811] ERROR - GET http://172.23.98.11:8094/api/index/index_default body: headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found
rest_auth: preparePerm, err: index not found auth: Administrator:password
[2018-08-26 17:09:53,189] - [rest_client:811] ERROR - DELETE http://172.23.98.11:8094/api/index/index_default body: headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found
rest_auth: preparePerm, err: index not found auth: Administrator:password
[2018-08-26 17:09:53,190] - [fts_base:904] INFO - Creating fulltext-index index_default on 172.23.98.11
[2018-08-26 17:09:53,190] - [rest_client:2699] INFO - {"params": {"store": {"mossStoreOptions": {}, "kvStoreName": "mossStore"}}, "name": "index_default", "planParams": {}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "uuid": ""}
[2018-08-26 17:09:53,213] - [rest_client:2706] INFO - Index index_default created
[2018-08-26 17:09:53,569] - [task:1248] INFO - Starting operation 'create' on Elastic Search ...
[2018-08-26 17:09:53,755] - [data_helper:295] INFO - creating direct client 172.23.97.230:11210 default
[2018-08-26 17:09:53,804] - [data_helper:295] INFO - creating direct client 172.23.98.11:11210 default
[2018-08-26 17:09:53,856] - [data_helper:295] INFO - creating direct client 172.23.98.19:11210 default
[2018-08-26 17:09:55,295] - [task:1282] INFO - 5000 documents bulk loaded into ES
[2018-08-26 17:09:55,372] - [task:1287] INFO - ES index count for 'es_index': 5000
[2018-08-26 17:09:55,586] - [rest_client:1925] INFO - http://172.23.97.230:8091/pools/default/buckets/default/stats?zoom=minute
[2018-08-26 17:09:55,609] - [fts_callable:97] INFO - Docs in bucket = 0, docs in FTS index 'index_default': 3065, docs in ES index: 5000
[2018-08-26 17:10:01,639] - [rest_client:1925] INFO - http://172.23.97.230:8091/pools/default/buckets/default/stats?zoom=minute
[2018-08-26 17:10:01,656] - [fts_callable:97] INFO - Docs in bucket = 5000, docs in FTS index 'index_default': 5000, docs in ES index: 5000
[2018-08-26 17:10:02,123] - [task:1316] INFO - ----------------------------------------------------- Query # 1 ----------------------------------------------------
[2018-08-26 17:10:02,131] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": true, "field": "salary", "max": 94590.0, "inclusive_min": true, "min": 92000.0}, "size": 10000000} on node: 172.23.98.19:
[2018-08-26 17:10:02,144] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:02,144] - [task:1344] INFO - FTS hits for query: {"inclusive_max": true, "field": "salary", "max": 94590.0, "inclusive_min": true, "min": 92000.0} is 124 (took 6.04486ms)
[2018-08-26 17:10:02,154] - [task:1354] INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"salary": {"gte": 92000.0, "lte": 94590.0}}}}}} on es_index is 124 (took 4ms)
[2018-08-26 17:10:02,155] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:02,155] - [task:1316] INFO - ----------------------------------------------------- Query # 2 ----------------------------------------------------
[2018-08-26 17:10:02,163] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"conjuncts": [{"field": "email", "match": "trista@mcdiabetes.com"}, {"field": "manages.reports", "match_phrase": "Basha"}, {"inclusive_max": true, "field": "mutated", "max": 9877, "inclusive_min": true, "min": 4}]}, "size": 10000000} on node: 172.23.98.19:
[2018-08-26 17:10:02,173] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:02,174] - [task:1344] INFO - FTS hits for query: {"conjuncts": [{"field": "email", "match": "trista@mcdiabetes.com"}, {"field": "manages.reports", "match_phrase": "Basha"}, {"inclusive_max": true, "field": "mutated", "max": 9877, "inclusive_min": true, "min": 4}]} is 0 (took 2.239884ms)
[2018-08-26 17:10:02,177] - [task:1354] INFO - ES hits for query: {"query": {"bool": {"must": [{"match": {"email": "trista@mcdiabetes.com"}}, {"match_phrase": {"manages.reports": "Basha"}}, {"filtered": {"filter": {"range": {"mutated": {"gte": 4, "lte": 9877}}}}}]}}} on es_index is 0 (took 1ms)
[2018-08-26 17:10:02,177] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:02,177] - [task:1316] INFO - ----------------------------------------------------- Query # 3 ----------------------------------------------------
[2018-08-26 17:10:02,184] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2018-08-26T17:10:01.674372", "start": "2010-04-21T21:41:00"}, "size": 10000000} on node: 172.23.98.11:
[2018-08-26 17:10:02,205] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:02,205] - [task:1344] INFO - FTS hits for query: {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2018-08-26T17:10:01.674372", "start": "2010-04-21T21:41:00"} is 525 (took 9.946178ms)
[2018-08-26 17:10:02,228] - [task:1354] INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"join_date": {"lt": "2018-08-26T17:10:01.674372", "gt": "2010-04-21T21:41:00"}}}}}} on es_index is 525 (took 9ms)
[2018-08-26 17:10:02,228] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:02,228] - [task:1316] INFO - ----------------------------------------------------- Query # 4 ----------------------------------------------------
[2018-08-26 17:10:02,235] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "dept", "match": "Pre-sales"}, {"field": "dept", "match": "HR"}]}, "size": 10000000} on node: 172.23.98.11:
[2018-08-26 17:10:02,255] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:02,255] - [task:1344] INFO - FTS hits for query: {"disjuncts": [{"field": "dept", "match": "Pre-sales"}, {"field": "dept", "match": "HR"}]} is 1390 (took 5.318124ms)
[2018-08-26 17:10:02,309] - [task:1354] INFO - ES hits for query: {"query": {"bool": {"should": [{"match": {"dept": "Pre-sales"}}, {"match": {"dept": "HR"}}]}}} on es_index is 1390 (took 21ms)
[2018-08-26 17:10:02,310] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:02,310] - [task:1316] INFO - ----------------------------------------------------- Query # 5 ----------------------------------------------------
[2018-08-26 17:10:02,318] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"conjuncts": [{"field": "name", "match": "Miller XII"}, {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2018-08-26T17:10:01.675001", "start": "1997-10-18T14:49:00"}]}, "size": 10000000} on node: 172.23.98.19:
[2018-08-26 17:10:02,331] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:02,331] - [task:1344] INFO - FTS hits for query: {"conjuncts": [{"field": "name", "match": "Miller XII"}, {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2018-08-26T17:10:01.675001", "start": "1997-10-18T14:49:00"}]} is 23 (took 6.279729ms)
[2018-08-26 17:10:02,336] - [task:1354] INFO - ES hits for query: {"query": {"bool": {"must": [{"match": {"name": "Miller XII"}}, {"filtered": {"filter": {"range": {"join_date": {"lt": "2018-08-26T17:10:01.675001", "gt": "1997-10-18T14:49:00"}}}}}]}}} on es_index is 23 (took 3ms)
[2018-08-26 17:10:02,336] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:02,336] - [task:1316] INFO - ----------------------------------------------------- Query # 6 ----------------------------------------------------
[2018-08-26 17:10:02,343] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "manages.reports", "regexp": "Ad*"}, "size": 10000000} on node: 172.23.97.230:
[2018-08-26 17:10:02,352] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:02,352] - [task:1344] INFO - FTS hits for query: {"field": "manages.reports", "regexp": "Ad*"} is 0 (took 2.16237ms)
[2018-08-26 17:10:02,356] - [task:1354] INFO - ES hits for query: {"query": {"regexp": {"manages.reports": "Ad*"}}} on es_index is 0 (took 1ms)
[2018-08-26 17:10:02,356] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:02,356] - [task:1316] INFO - ----------------------------------------------------- Query # 7 ----------------------------------------------------
[2018-08-26 17:10:02,364] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2018-08-26T17:10:01.675560", "start": "2003-02-16T02:59:00"}, "size": 10000000} on node: 172.23.98.19:
[2018-08-26 17:10:02,387] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:02,388] - [task:1344] INFO - FTS hits for query: {"inclusive_start": true, "field": "join_date", "inclusive_end": true, "end": "2018-08-26T17:10:01.675560", "start": "2003-02-16T02:59:00"} is 1071 (took 6.954649ms)
[2018-08-26 17:10:02,426] - [task:1354] INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"join_date": {"gte": "2003-02-16T02:59:00", "lte": "2018-08-26T17:10:01.675560"}}}}}} on es_index is 1071 (took 14ms)
[2018-08-26 17:10:02,427] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:02,427] - [task:1316] INFO - ----------------------------------------------------- Query # 8 ----------------------------------------------------
[2018-08-26 17:10:02,434] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "name", "match": "Kacila"}, {"field": "name", "match": "Quella Cook"}, {"field": "type", "match": "emp"}]}, "size": 10000000} on node: 172.23.98.19:
[2018-08-26 17:10:02,485] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:02,486] - [task:1344] INFO - FTS hits for query: {"disjuncts": [{"field": "name", "match": "Kacila"}, {"field": "name", "match": "Quella Cook"}, {"field": "type", "match": "emp"}]} is 5000 (took 14.028148ms)
[2018-08-26 17:10:02,666] - [task:1354] INFO - ES hits for query: {"query": {"bool": {"should": [{"match": {"name": "Kacila"}}, {"match": {"name": "Quella Cook"}}, {"match": {"type": "emp"}}]}}} on es_index is 5000 (took 60ms)
[2018-08-26 17:10:02,670] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:02,670] - [task:1316] INFO - ----------------------------------------------------- Query # 9 ----------------------------------------------------
[2018-08-26 17:10:02,680] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "languages_known", "wildcard": "Qu?chua"}, "size": 10000000} on node: 172.23.98.19:
[2018-08-26 17:10:02,688] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:02,688] - [task:1344] INFO - FTS hits for query: {"field": "languages_known", "wildcard": "Qu?chua"} is 0 (took 1.469655ms)
[2018-08-26 17:10:02,692] - [task:1354] INFO - ES hits for query: {"query": {"wildcard": {"languages_known": "Qu?chua"}}} on es_index is 0 (took 2ms)
[2018-08-26 17:10:02,692] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:02,692] - [task:1316] INFO - ----------------------------------------------------- Query # 10 ----------------------------------------------------
[2018-08-26 17:10:02,699] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "match_phrase": "Riona Johnson"}, "size": 10000000} on node: 172.23.97.230:
[2018-08-26 17:10:02,705] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:02,705] - [task:1344] INFO - FTS hits for query: {"field": "name", "match_phrase": "Riona Johnson"} is 0 (took 1.145643ms)
[2018-08-26 17:10:02,708] - [task:1354] INFO - ES hits for query: {"query": {"match_phrase": {"name": "Riona Johnson"}}} on es_index is 0 (took 1ms)
[2018-08-26 17:10:02,708] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:02,709] - [task:1316] INFO - ----------------------------------------------------- Query # 11 ----------------------------------------------------
[2018-08-26 17:10:02,715] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2018-08-26T17:10:01.676554", "start": "1954-09-04T03:02:00"}, "size": 10000000} on node: 172.23.97.230:
[2018-08-26 17:10:02,775] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:02,775] - [task:1344] INFO - FTS hits for query: {"inclusive_start": false, "field": "join_date", "inclusive_end": false, "end": "2018-08-26T17:10:01.676554", "start": "1954-09-04T03:02:00"} is 4628 (took 21.551446ms)
[2018-08-26 17:10:02,934] - [task:1354] INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"join_date": {"lt": "2018-08-26T17:10:01.676554", "gt": "1954-09-04T03:02:00"}}}}}} on es_index is 4628 (took 50ms)
[2018-08-26 17:10:02,936] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:02,937] - [task:1316] INFO - ----------------------------------------------------- Query # 12 ----------------------------------------------------
[2018-08-26 17:10:02,945] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": true, "field": "salary", "max": 107914.0, "inclusive_min": true, "min": 104000.0}, "size": 10000000} on node: 172.23.98.19:
[2018-08-26 17:10:02,959] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:02,959] - [task:1344] INFO - FTS hits for query: {"inclusive_max": true, "field": "salary", "max": 107914.0, "inclusive_min": true, "min": 104000.0} is 195 (took 4.725323ms)
[2018-08-26 17:10:02,970] - [task:1354] INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"salary": {"gte": 104000.0, "lte": 107914.0}}}}}} on es_index is 195 (took 5ms)
[2018-08-26 17:10:02,971] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:02,971] - [task:1316] INFO - ----------------------------------------------------- Query # 13 ----------------------------------------------------
[2018-08-26 17:10:02,980] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "dept", "match": "Support"}, {"field": "email", "match": "riona@mcdiabetes.com"}, {"field": "manages.reports", "match": "Gallia Kerry"}]}, "size": 10000000} on node: 172.23.97.230:
[2018-08-26 17:10:03,036] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:03,036] - [task:1344] INFO - FTS hits for query: {"disjuncts": [{"field": "dept", "match": "Support"}, {"field": "email", "match": "riona@mcdiabetes.com"}, {"field": "manages.reports", "match": "Gallia Kerry"}]} is 5000 (took 16.558456ms)
[2018-08-26 17:10:03,185] - [task:1354] INFO - ES hits for query: {"query": {"bool": {"should": [{"match": {"dept": "Support"}}, {"match": {"email": "riona@mcdiabetes.com"}}, {"match": {"manages.reports": "Gallia Kerry"}}]}}} on es_index is 5000 (took 38ms)
[2018-08-26 17:10:03,188] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:03,189] - [task:1316] INFO - ----------------------------------------------------- Query # 14 ----------------------------------------------------
[2018-08-26 17:10:03,196] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "manages.reports", "regexp": "An.*"}, "size": 10000000} on node: 172.23.98.19:
[2018-08-26 17:10:03,207] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:03,207] - [task:1344] INFO - FTS hits for query: {"field": "manages.reports", "regexp": "An.*"} is 0 (took 3.825168ms)
[2018-08-26 17:10:03,210] - [task:1354] INFO - ES hits for query: {"query": {"regexp": {"manages.reports": "An.*"}}} on es_index is 0 (took 2ms)
[2018-08-26 17:10:03,211] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:03,211] - [task:1316] INFO - ----------------------------------------------------- Query # 15 ----------------------------------------------------
[2018-08-26 17:10:03,218] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "languages_known", "prefix": "A"}, "size": 10000000} on node: 172.23.98.19:
[2018-08-26 17:10:03,225] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:03,225] - [task:1344] INFO - FTS hits for query: {"field": "languages_known", "prefix": "A"} is 0 (took 1.409264ms)
[2018-08-26 17:10:03,227] - [task:1354] INFO - ES hits for query: {"query": {"prefix": {"languages_known": "A"}}} on es_index is 0 (took 1ms)
[2018-08-26 17:10:03,228] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:03,228] - [task:1316] INFO - ----------------------------------------------------- Query # 16 ----------------------------------------------------
[2018-08-26 17:10:03,235] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "email", "match": "desdomna@mcdiabetes.com"}, "size": 10000000} on node: 172.23.98.11:
[2018-08-26 17:10:04,166] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:04,167] - [task:1344] INFO - FTS hits for query: {"field": "email", "match": "desdomna@mcdiabetes.com"} is 5000 (took 21.080552ms)
[2018-08-26 17:10:04,329] - [task:1354] INFO - ES hits for query: {"query": {"match": {"email": "desdomna@mcdiabetes.com"}}} on es_index is 5000 (took 55ms)
[2018-08-26 17:10:04,331] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:04,332] - [task:1316] INFO - ----------------------------------------------------- Query # 17 ----------------------------------------------------
[2018-08-26 17:10:04,344] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "name", "match_phrase": "Alvita Ann"}, "size": 10000000} on node: 172.23.98.19:
[2018-08-26 17:10:04,352] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:04,352] - [task:1344] INFO - FTS hits for query: {"field": "name", "match_phrase": "Alvita Ann"} is 1 (took 1.453927ms)
[2018-08-26 17:10:04,356] - [task:1354] INFO - ES hits for query: {"query": {"match_phrase": {"name": "Alvita Ann"}}} on es_index is 1 (took 1ms)
[2018-08-26 17:10:04,356] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:04,356] - [task:1316] INFO - ----------------------------------------------------- Query # 18 ----------------------------------------------------
[2018-08-26 17:10:04,363] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"inclusive_max": true, "field": "mutated", "max": 6347, "inclusive_min": true, "min": 4}, "size": 10000000} on node: 172.23.97.230:
[2018-08-26 17:10:04,370] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:04,370] - [task:1344] INFO - FTS hits for query: {"inclusive_max": true, "field": "mutated", "max": 6347, "inclusive_min": true, "min": 4} is 0 (took 1.171678ms)
[2018-08-26 17:10:04,374] - [task:1354] INFO - ES hits for query: {"query": {"filtered": {"filter": {"range": {"mutated": {"gte": 4, "lte": 6347}}}}}} on es_index is 0 (took 1ms)
[2018-08-26 17:10:04,374] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:04,374] - [task:1316] INFO - ----------------------------------------------------- Query # 19 ----------------------------------------------------
[2018-08-26 17:10:04,381] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"disjuncts": [{"field": "name", "match": "Desdomna"}, {"field": "email", "match": "araceli@mcdiabetes.com"}, {"field": "name", "match": "Sebastián"}]}, "size": 10000000} on node: 172.23.98.11:
[2018-08-26 17:10:04,441] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:04,441] - [task:1344] INFO - FTS hits for query: {"disjuncts": [{"field": "name", "match": "Desdomna"}, {"field": "email", "match": "araceli@mcdiabetes.com"}, {"field": "name", "match": "Sebastián"}]} is 5000 (took 17.496405ms)
[2018-08-26 17:10:04,590] - [task:1354] INFO - ES hits for query: {"query": {"bool": {"should": [{"match": {"name": "Desdomna"}}, {"match": {"email": "araceli@mcdiabetes.com"}}, {"match": {"name": "Sebastián"}}]}}} on es_index is 5000 (took 36ms)
[2018-08-26 17:10:04,593] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:04,593] - [task:1316] INFO - ----------------------------------------------------- Query # 20 ----------------------------------------------------
[2018-08-26 17:10:04,601] - [fts_base:2061] INFO - Running query {"indexName": "index_default", "from": 0, "fields": [], "explain": false, "ctl": {"timeout": 60000, "consistency": {"vectors": {}, "level": ""}}, "query": {"field": "type", "match": "wiki"}, "size": 10000000} on node: 172.23.98.11:
[2018-08-26 17:10:04,610] - [task:1320] INFO - Status: {u'successful': 6, u'failed': 0, u'total': 6}
[2018-08-26 17:10:04,610] - [task:1344] INFO - FTS hits for query: {"field": "type", "match": "wiki"} is 0 (took 3.404752ms)
[2018-08-26 17:10:04,614] - [task:1354] INFO - ES hits for query: {"query": {"match": {"type": "wiki"}}} on es_index is 0 (took 1ms)
[2018-08-26 17:10:04,614] - [task:1363] INFO - SUCCESS: Docs returned by FTS = docs returned by ES, doc_ids verified
[2018-08-26 17:10:04,615] - [fts_callable:219] INFO - SUCCESS: 20 out of 20 queries passed
[2018-08-26 17:10:04,622] - [rest_client:2026] INFO - Node versions in cluster [u'5.0.0-3519-enterprise', u'5.0.0-3519-enterprise', u'5.0.0-3519-enterprise']
[2018-08-26 17:10:04,629] - [basetestcase:2295] INFO - list of kv nodes in cluster: [ip:172.23.98.19 port:8091 ssh_username:root, ip:172.23.97.230 port:8091 ssh_username:root, ip:172.23.98.11 port:8091 ssh_username:root]
[2018-08-26 17:10:05,174] - [data_helper:295] INFO - creating direct client 172.23.97.230:11210 default
[2018-08-26 17:10:05,217] - [task:623] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.97.230:8091',default bucket
[2018-08-26 17:10:05,225] - [data_helper:295] INFO - creating direct client 172.23.98.11:11210 default
[2018-08-26 17:10:05,271] - [task:623] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.98.11:8091',default bucket
[2018-08-26 17:10:05,283] - [data_helper:295] INFO - creating direct client 172.23.98.19:11210 default
[2018-08-26 17:10:05,328] - [task:623] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.98.19:8091',default bucket
[2018-08-26 17:10:05,340] - [upgrade_tests:149] INFO -
*** Start upgrade cluster ***
[2018-08-26 17:10:05,340] - [upgrade_tests:289] INFO - upgrade_event
[2018-08-26 17:10:05,341] - [upgrade_tests:809] INFO - online_upgrade
[2018-08-26 17:10:05,341] - [newupgradebasetest:663] INFO - No need to do DCP rebalance upgrade
[2018-08-26 17:10:05,341] - [basetestcase:489] INFO - sleep for 15 secs. Pre-setup of old version is done. Wait for online upgrade to 6.0.0-1564 version ...
[2018-08-26 17:10:20,357] - [upgrade_tests:824] INFO - online_upgrade_swap_rebalance
[2018-08-26 17:10:20,362] - [upgrade_tests:851] INFO - ****************************************
[2018-08-26 17:10:20,362] - [upgrade_tests:852] INFO - cluster nodes = [ip:172.23.98.19 port:8091 ssh_username:root, ip:172.23.97.230 port:8091 ssh_username:root, ip:172.23.98.11 port:8091 ssh_username:root]
[2018-08-26 17:10:20,367] - [upgrade_tests:853] INFO - cluster service map = {'172.23.98.19:8091': [u'fts', u'kv'], '172.23.97.230:8091': [u'fts', u'kv'], '172.23.98.11:8091': [u'fts', u'kv']}
[2018-08-26 17:10:20,371] - [rest_client:2016] INFO - Node version in cluster 5.0.0-3519-enterprise
[2018-08-26 17:10:20,371] - [upgrade_tests:854] INFO - cluster version map = 5.0.0-3519-enterprise
[2018-08-26 17:10:20,371] - [upgrade_tests:855] INFO - to include in cluster = [ip:172.23.98.214 port:8091 ssh_username:root]
[2018-08-26 17:10:20,371] - [upgrade_tests:856] INFO - to exclude from cluster = [ip:172.23.98.19 port:8091 ssh_username:root]
[2018-08-26 17:10:20,372] - [upgrade_tests:857] INFO - ****************************************
[2018-08-26 17:10:20,378] - [newupgradebasetest:195] INFO - will install 6.0.0-1564 on ['172.23.98.214']
[2018-08-26 17:10:20,380] - [remote_util:232] INFO - connecting to 172.23.98.214 with username:root password:couchbase ssh_key:
[2018-08-26 17:10:20,477] - [remote_util:266] INFO - Connected to 172.23.98.214
[2018-08-26 17:10:20,688] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:10:20,975] - [remote_util:2404] INFO - 172.23.98.214 *****In couchbase uninstall****
[2018-08-26 17:10:20,975] - [remote_util:2412] INFO - CentOS
[2018-08-26 17:10:21,084] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: killall -9 rpm
[2018-08-26 17:10:21,110] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:10:21,111] - [remote_util:2934] ERROR - rpm: no process found
[2018-08-26 17:10:21,111] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: rm -f /var/lib/rpm/.rpm.lock
[2018-08-26 17:10:21,188] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:10:21,294] - [remote_util:2692] INFO - running rpm -e to remove couchbase-server
[2018-08-26 17:10:21,294] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: rpm -e couchbase-server
[2018-08-26 17:10:24,506] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:10:24,506] - [remote_util:2934] ERROR -
**** Uninstalling on server: 172.23.98.214 ****
[2018-08-26 17:10:24,506] - [remote_util:2934] ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory
[2018-08-26 17:10:24,507] - [remote_util:2934] ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave
[2018-08-26 17:10:24,507] - [remote_util:2934] ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave
[2018-08-26 17:10:24,507] - [remote_util:2934] ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave
[2018-08-26 17:10:24,507] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: pkill -u couchbase
[2018-08-26 17:10:24,526] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:10:24,527] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: ps -ef | grep couchbase
[2018-08-26 17:10:24,617] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:10:24,617] - [remote_util:2972] INFO - root 8492 8324 0 17:10 ? 00:00:00 bash -c ps -ef | grep couchbase
[2018-08-26 17:10:24,618] - [remote_util:2972] INFO - root 8498 8492 0 17:10 ? 00:00:00 grep couchbase
[2018-08-26 17:10:26,143] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: kill -9 $(ps aux | pgrep 'memcached')
[2018-08-26 17:10:26,226] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:10:26,226] - [remote_util:2934] ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
[2018-08-26 17:10:26,226] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: ipcrm
[2018-08-26 17:10:26,300] - [remote_util:3174] INFO - command executed successfully
thread uninstaller-thread-172.23.98.214 finished
[2018-08-26 17:10:26,303] - [remote_util:232] INFO - connecting to 172.23.98.214 with username:root password:couchbase ssh_key:
[2018-08-26 17:10:26,432] - [remote_util:266] INFO - Connected to 172.23.98.214
[2018-08-26 17:10:26,642] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:10:27,009] - [install:629] INFO - ********CouchbaseServerInstaller:install
[2018-08-26 17:10:27,011] - [remote_util:232] INFO - connecting to 172.23.98.214 with username:root password:couchbase ssh_key:
[2018-08-26 17:10:27,127] - [remote_util:266] INFO - Connected to 172.23.98.214
[2018-08-26 17:10:27,340] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
*** OS version of this server 172.23.98.214 is CentOS 7 ***
[2018-08-26 17:10:27,631] - [remote_util:232] INFO - connecting to 172.23.98.214 with username:root password:couchbase ssh_key:
[2018-08-26 17:10:27,728] - [remote_util:266] INFO - Connected to 172.23.98.214
[2018-08-26 17:10:27,920] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:10:28,255] - [remote_util:792] INFO - Check if url http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/1564/couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm is ok
[2018-08-26 17:10:28,258] - [remote_util:795] INFO - This url http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/1564/couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm is live
[2018-08-26 17:10:28,259] - [remote_util:232] INFO - connecting to 172.23.98.214 with username:root password:couchbase ssh_key:
[2018-08-26 17:10:28,376] - [remote_util:266] INFO - Connected to 172.23.98.214
[2018-08-26 17:10:28,563] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:10:28,875] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: /sbin/iptables -F
[2018-08-26 17:10:28,951] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:10:28,952] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: /sbin/iptables -t nat -F
[2018-08-26 17:10:29,027] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:10:29,027] - [remote_util:294] INFO - Connect to node: 172.23.98.214 as user: root
[2018-08-26 17:10:29,124] - [remote_util:317] INFO - Connected to 172.23.98.214 as root
[2018-08-26 17:10:29,124] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D
[2018-08-26 17:10:29,250] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:10:29,250] - [remote_util:2934] ERROR - mv: cannot stat ‘core.*’: No such file or directory
[2018-08-26 17:10:29,251] - [remote_util:2934] ERROR - rm: cannot remove ‘atop.d’: Is a directory
[2018-08-26 17:10:29,251] - [remote_util:2934] ERROR - rm: cannot remove ‘cb_4XxT’: Is a directory
[2018-08-26 17:10:29,251] - [remote_util:2934] ERROR - rm: cannot remove ‘hsperfdata_couchbase’: Is a directory
[2018-08-26 17:10:29,251] - [remote_util:2934] ERROR - rm: cannot remove ‘systemd-private-dcff3f99cbc245528bd2fce43f9aa0bf-ntpd.service-GOgZPf’: Is a directory
[2018-08-26 17:10:29,251] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/1564/couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm;cd /tmp;ls -lh
[2018-08-26 17:10:34,027] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:10:34,027] - [remote_util:2972] INFO - total 243M
[2018-08-26 17:10:34,028] - [remote_util:2972] INFO - drwx------. 2 root root 22 Aug 17 00:21 atop.d
[2018-08-26 17:10:34,029] - [remote_util:2972] INFO - -rw-r--r--. 1 root root 243M Aug 24 12:20 couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm
[2018-08-26 17:10:34,030] - [remote_util:2972] INFO - drwxr-x---. 2 couchbase couchbase 6 Aug 16 19:36 hsperfdata_couchbase
[2018-08-26 17:10:34,030] - [remote_util:2972] INFO - drwx------. 3 root root 16 Aug 17 00:21 systemd-private-dcff3f99cbc245528bd2fce43f9aa0bf-ntpd.service-GOgZPf
[2018-08-26 17:10:34,111] - [remote_util:1832] INFO - *****install server ***
[2018-08-26 17:10:34,112] - [remote_util:1847] INFO - deliverable_type : rpm
[2018-08-26 17:10:34,112] - [remote_util:1888] INFO - /tmp/couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm or /tmp/
[2018-08-26 17:10:34,112] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: /sbin/sysctl vm.swappiness=0
[2018-08-26 17:10:34,188] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:10:34,188] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: cat /etc/redhat-release
[2018-08-26 17:10:34,263] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:10:34,263] - [remote_util:2972] INFO - CentOS Linux release 7.2.1511 (Core)
[2018-08-26 17:10:34,264] - [remote_util:4480] INFO - no need to install pkgconfig
[2018-08-26 17:11:08,005] - [remote_util:1679] ERROR - Directory at /tmp DOES exist. Fx returns True
[2018-08-26 17:11:08,099] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server
[2018-08-26 17:11:08,152] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:11:08,243] - [remote_util:3582] INFO - Running systemd command on this server
[2018-08-26 17:11:08,243] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: systemctl start couchbase-server.service
[2018-08-26 17:11:08,289] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:11:08,365] - [install:731] INFO - wait 5 seconds for Couchbase server to start
thread installer-thread-172.23.98.214 finished
[2018-08-26 17:11:13,380] - [remote_util:232] INFO - connecting to 172.23.98.214 with username:root password:couchbase ssh_key:
[2018-08-26 17:11:13,546] - [remote_util:266] INFO - Connected to 172.23.98.214
[2018-08-26 17:11:13,730] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:11:14,144] - [remote_util:749] INFO - 172.23.98.214 **** The version file /opt/couchbase/ VERSION.txt exists
[2018-08-26 17:11:14,145] - [rest_client:816] ERROR - socket error while connecting to http://172.23.98.214:8091/nodes/self error [Errno 111] Connection refused
[2018-08-26 17:11:17,149] - [rest_client:816] ERROR - socket error while connecting to http://172.23.98.214:8091/nodes/self error [Errno 111] Connection refused
[2018-08-26 17:11:23,184] - [rest_client:2016] INFO - Node version in cluster 6.0.0-1564-enterprise
[2018-08-26 17:11:23,184] - [remote_util:4597] INFO - ***** set NS_SERVER_CBAUTH env in linux *****
[2018-08-26 17:11:23,400] - [remote_util:4483] INFO - check if man installed on vm?
[2018-08-26 17:11:23,400] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: rpm -qa | grep 'man-db'
[2018-08-26 17:11:23,995] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:11:23,995] - [remote_util:4344] INFO -
---> Run command unzip to check if it is ready on server 172.23.98.214
[2018-08-26 17:11:23,996] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: unzip
[2018-08-26 17:11:24,072] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:11:24,072] - [remote_util:4376] INFO - unzip command is ready
[2018-08-26 17:11:24,072] - [remote_util:810] INFO - Check if ntp is installed
[2018-08-26 17:11:24,073] - [remote_util:813] INFO -
This OS version centos 7
[2018-08-26 17:11:24,073] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: systemctl status ntpd
[2018-08-26 17:11:24,164] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:11:24,164] - [remote_util:3137] INFO - running command.raw on 172.23.98.214: date
[2018-08-26 17:11:24,241] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:11:24,373] - [remote_util:882] INFO -
['Sun Aug 26 17:11:24 PDT 2018'] IP: 172.23.98.214
thread initializer-thread-172.23.98.214 finished
[2018-08-26 17:11:24,488] - [remote_util:232] INFO - connecting to 172.23.98.214 with username:root password:couchbase ssh_key:
[2018-08-26 17:11:24,585] - [remote_util:266] INFO - Connected to 172.23.98.214
[2018-08-26 17:11:24,771] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:11:25,089] - [remote_util:232] INFO - connecting to 172.23.98.214 with username:root password:couchbase ssh_key:
[2018-08-26 17:11:25,185] - [remote_util:266] INFO - Connected to 172.23.98.214
[2018-08-26 17:11:25,396] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:11:25,823] - [remote_util:749] INFO - 172.23.98.214 **** The version file /opt/couchbase/ VERSION.txt exists
[2018-08-26 17:11:25,823] - [basetestcase:489] INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ...
[2018-08-26 17:11:30,952] - [basetestcase:489] INFO - sleep for 10 secs. Wait for ns server is ready ...
[2018-08-26 17:11:40,962] - [upgrade_tests:200] INFO - record vbucket for the bucket default
[2018-08-26 17:11:41,437] - [rest_client:2026] INFO - Node versions in cluster [u'5.0.0-3519-enterprise', u'5.0.0-3519-enterprise', u'5.0.0-3519-enterprise']
[2018-08-26 17:11:41,474] - [task:431] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:11:41,474] - [task:446] INFO - adding node 172.23.98.214:8091 to cluster
[2018-08-26 17:11:41,474] - [rest_client:1244] INFO - adding remote node @172.23.98.214:8091 to this cluster @172.23.98.19:8091
[2018-08-26 17:11:43,610] - [rest_client:1462] INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.98.19', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.98.19,ns_1@172.23.98.214,ns_1@172.23.97.230,ns_1@172.23.98.11'}
[2018-08-26 17:11:53,673] - [rest_client:1467] INFO - rebalance operation started
[2018-08-26 17:11:53,673] - [task:497] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:11:53,715] - [rest_client:1615] INFO - rebalance percentage : 0.00 %
[2018-08-26 17:11:53,715] - [task:512] INFO - Rebalance - status: running, progress: 0.00%
[2018-08-26 17:12:03,731] - [task:497] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:12:03,781] - [rest_client:1615] INFO - rebalance percentage : 40.03 %
[2018-08-26 17:12:03,781] - [task:512] INFO - Rebalance - status: running, progress: 40.03%
[2018-08-26 17:12:13,797] - [task:497] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:12:13,844] - [rest_client:1615] INFO - rebalance percentage : 50.00 %
[2018-08-26 17:12:13,844] - [task:512] INFO - Rebalance - status: running, progress: 50.00%
[2018-08-26 17:12:23,860] - [task:497] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:12:23,898] - [rest_client:1615] INFO - rebalance percentage : 150.00 %
[2018-08-26 17:12:23,898] - [task:512] INFO - Rebalance - status: running, progress: 150.00%
[2018-08-26 17:12:33,914] - [task:497] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:12:33,955] - [task:512] INFO - Rebalance - status: none, progress: 100.00%
[2018-08-26 17:12:33,958] - [rest_client:811] ERROR - GET http://172.23.98.19:8091/pools/default body: headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password
http://172.23.98.19:8091/pools/default with status False: unknown pool
[2018-08-26 17:12:33,967] - [task:571] INFO - rebalancing was completed with progress: 100% in 40.293211937 sec
[2018-08-26 17:12:33,967] - [upgrade_tests:200] INFO - record vbucket for the bucket default
[2018-08-26 17:12:34,022] - [newupgradebasetest:607] INFO - Checking active_vb on nodes that remain in cluster...
[2018-08-26 17:12:34,022] - [newupgradebasetest:613] INFO - active_vb vbuckets were not suffled
[2018-08-26 17:12:34,022] - [newupgradebasetest:614] INFO - Checking in-out nodes...
[2018-08-26 17:12:34,022] - [newupgradebasetest:607] INFO - Checking replica_vb on nodes that remain in cluster...
[2018-08-26 17:12:34,022] - [newupgradebasetest:613] INFO - replica_vb vbuckets were not suffled
[2018-08-26 17:12:34,022] - [newupgradebasetest:614] INFO - Checking in-out nodes...
[2018-08-26 17:12:34,026] - [upgrade_tests:851] INFO - ****************************************
[2018-08-26 17:12:34,027] - [upgrade_tests:852] INFO - cluster nodes = [ip:172.23.98.11 port:8091 ssh_username:root, ip:172.23.98.214 port:8091 ssh_username:root, ip:172.23.97.230 port:8091 ssh_username:root]
[2018-08-26 17:12:34,031] - [upgrade_tests:853] INFO - cluster service map = {'172.23.98.214:8091': [u'fts', u'kv'], '172.23.97.230:8091': [u'fts', u'kv'], '172.23.98.11:8091': [u'fts', u'kv']}
[2018-08-26 17:12:34,035] - [rest_client:2016] INFO - Node version in cluster 5.0.0-3519-enterprise
[2018-08-26 17:12:34,035] - [upgrade_tests:854] INFO - cluster version map = 5.0.0-3519-enterprise
[2018-08-26 17:12:34,035] - [upgrade_tests:855] INFO - to include in cluster = [ip:172.23.98.19 port:8091 ssh_username:root]
[2018-08-26 17:12:34,035] - [upgrade_tests:856] INFO - to exclude from cluster = [ip:172.23.98.11 port:8091 ssh_username:root]
[2018-08-26 17:12:34,035] - [upgrade_tests:857] INFO - ****************************************
[2018-08-26 17:12:34,042] - [newupgradebasetest:195] INFO - will install 6.0.0-1564 on ['172.23.98.19']
[2018-08-26 17:12:34,044] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:12:34,143] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:12:34,413] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:12:34,715] - [remote_util:2404] INFO - 172.23.98.19 *****In couchbase uninstall****
[2018-08-26 17:12:34,716] - [remote_util:2412] INFO - CentOS
[2018-08-26 17:12:34,825] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: killall -9 rpm
[2018-08-26 17:12:34,853] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:12:34,854] - [remote_util:2934] ERROR - rpm: no process found
[2018-08-26 17:12:34,854] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: rm -f /var/lib/rpm/.rpm.lock
[2018-08-26 17:12:34,927] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:12:35,031] - [remote_util:2692] INFO - running rpm -e to remove couchbase-server
[2018-08-26 17:12:35,032] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: rpm -e couchbase-server
[2018-08-26 17:12:36,788] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:12:36,788] - [remote_util:2934] ERROR -
**** Uninstalling on server: 172.23.98.19 ****
[2018-08-26 17:12:36,788] - [remote_util:2934] ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory
[2018-08-26 17:12:36,788] - [remote_util:2934] ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave
[2018-08-26 17:12:36,788] - [remote_util:2934] ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave
[2018-08-26 17:12:36,788] - [remote_util:2934] ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave
[2018-08-26 17:12:36,788] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: pkill -u couchbase
[2018-08-26 17:12:36,806] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:12:36,806] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: ps -ef | grep couchbase
[2018-08-26 17:12:36,892] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:12:36,893] - [remote_util:2972] INFO - root 9329 9185 0 17:12 ? 00:00:00 bash -c ps -ef | grep couchbase
[2018-08-26 17:12:36,893] - [remote_util:2972] INFO - root 9335 9329 0 17:12 ? 00:00:00 grep couchbase
[2018-08-26 17:12:38,467] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: kill -9 $(ps aux | pgrep 'memcached')
[2018-08-26 17:12:38,549] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:12:38,549] - [remote_util:2934] ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
[2018-08-26 17:12:38,549] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: ipcrm
[2018-08-26 17:12:38,623] - [remote_util:3174] INFO - command executed successfully
thread uninstaller-thread-172.23.98.19 finished
[2018-08-26 17:12:38,626] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:12:38,723] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:12:38,972] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:12:39,324] - [install:629] INFO - ********CouchbaseServerInstaller:install
[2018-08-26 17:12:39,326] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:12:39,423] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:12:39,637] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
*** OS version of this server 172.23.98.19 is CentOS 7 ***
[2018-08-26 17:12:39,986] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:12:40,083] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:12:40,355] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:12:40,663] - [remote_util:792] INFO - Check if url http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/1564/couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm is ok
[2018-08-26 17:12:40,668] - [remote_util:795] INFO - This url http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/1564/couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm is live
[2018-08-26 17:12:40,669] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:12:40,749] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:12:40,993] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:12:41,263] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: /sbin/iptables -F
[2018-08-26 17:12:41,337] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:12:41,338] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: /sbin/iptables -t nat -F
[2018-08-26 17:12:41,411] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:12:41,412] - [remote_util:294] INFO - Connect to node: 172.23.98.19 as user: root
[2018-08-26 17:12:41,493] - [remote_util:317] INFO - Connected to 172.23.98.19 as root
[2018-08-26 17:12:41,493] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D
[2018-08-26 17:12:41,699] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:12:41,699] - [remote_util:2934] ERROR - mv: cannot stat ‘couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm’: No such file or directory
[2018-08-26 17:12:41,700] - [remote_util:2934] ERROR - mv: cannot stat ‘core.*’: No such file or directory
[2018-08-26 17:12:41,700] - [remote_util:2934] ERROR - rm: cannot remove ‘cb_b2iZ’: Is a directory
[2018-08-26 17:12:41,700] - [remote_util:2934] ERROR - rm: cannot remove ‘entbackup’: Is a directory
[2018-08-26 17:12:41,700] - [remote_util:2934] ERROR - rm: cannot remove ‘systemd-private-8dced1e9577f43b3a692103e0615fe24-ntpd.service-Ah1cAQ’: Is a directory
[2018-08-26 17:12:41,700] - [remote_util:2934] ERROR - mv: cannot stat ‘cb_b2iZ/*’: No such file or directory
[2018-08-26 17:12:41,700] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: cd /tmp;wget -q -O couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm http://172.23.120.24/builds/latestbuilds/couchbase-server/alice/1564/couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm;cd /tmp;ls -lh
[2018-08-26 17:12:43,968] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:12:43,968] - [remote_util:2972] INFO - total 243M
[2018-08-26 17:12:43,969] - [remote_util:2972] INFO - -rw-r--r--. 1 root root 243M Aug 24 12:20 couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm
[2018-08-26 17:12:43,969] - [remote_util:2972] INFO - drwxrwx---. 4 root root 44 Aug 25 09:00 entbackup
[2018-08-26 17:12:43,969] - [remote_util:2972] INFO - drwx------. 3 root root 16 Aug 26 06:27 systemd-private-8dced1e9577f43b3a692103e0615fe24-ntpd.service-Ah1cAQ
[2018-08-26 17:12:44,048] - [remote_util:1832] INFO - *****install server ***
[2018-08-26 17:12:44,048] - [remote_util:1847] INFO - deliverable_type : rpm
[2018-08-26 17:12:44,048] - [remote_util:1888] INFO - /tmp/couchbase-server-enterprise-6.0.0-1564-centos7.x86_64.rpm or /tmp/
[2018-08-26 17:12:44,048] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: /sbin/sysctl vm.swappiness=0
[2018-08-26 17:12:44,124] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:12:44,125] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: cat /etc/redhat-release
[2018-08-26 17:12:44,198] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:12:44,198] - [remote_util:2972] INFO - CentOS Linux release 7.3.1611 (Core)
[2018-08-26 17:12:44,198] - [remote_util:4480] INFO - no need to install pkgconfig
[2018-08-26 17:13:09,846] - [remote_util:1679] ERROR - Directory at /tmp DOES exist. Fx returns True
[2018-08-26 17:13:09,934] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server
[2018-08-26 17:13:10,007] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:13:10,114] - [remote_util:3582] INFO - Running systemd command on this server
[2018-08-26 17:13:10,116] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: systemctl start couchbase-server.service
[2018-08-26 17:13:10,150] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:13:10,225] - [install:731] INFO - wait 5 seconds for Couchbase server to start
thread installer-thread-172.23.98.19 finished
[2018-08-26 17:13:15,234] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:13:15,331] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:13:15,602] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:13:16,032] - [remote_util:749] INFO - 172.23.98.19 **** The version file /opt/couchbase/ VERSION.txt exists
[2018-08-26 17:13:16,033] - [rest_client:816] ERROR - socket error while connecting to http://172.23.98.19:8091/nodes/self error [Errno 111] Connection refused
[2018-08-26 17:13:22,064] - [rest_client:2016] INFO - Node version in cluster 6.0.0-1564-enterprise
[2018-08-26 17:13:22,064] - [remote_util:4597] INFO - ***** set NS_SERVER_CBAUTH env in linux *****
[2018-08-26 17:13:22,269] - [remote_util:4483] INFO - check if man installed on vm?
[2018-08-26 17:13:22,270] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: rpm -qa | grep 'man-db'
[2018-08-26 17:13:22,877] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:13:22,877] - [remote_util:4344] INFO -
---> Run command unzip to check if it is ready on server 172.23.98.19
[2018-08-26 17:13:22,877] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: unzip
[2018-08-26 17:13:22,952] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:13:22,952] - [remote_util:4376] INFO - unzip command is ready
[2018-08-26 17:13:22,952] - [remote_util:810] INFO - Check if ntp is installed
[2018-08-26 17:13:22,953] - [remote_util:813] INFO -
This OS version centos 7
[2018-08-26 17:13:22,953] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: systemctl status ntpd
[2018-08-26 17:13:23,042] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:13:23,042] - [remote_util:3137] INFO - running command.raw on 172.23.98.19: date
[2018-08-26 17:13:23,116] - [remote_util:3174] INFO - command executed successfully
[2018-08-26 17:13:23,283] - [remote_util:882] INFO -
['Sun Aug 26 17:13:23 PDT 2018'] IP: 172.23.98.19
thread initializer-thread-172.23.98.19 finished
[2018-08-26 17:13:23,399] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:13:23,480] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:13:23,725] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:13:24,027] - [remote_util:232] INFO - connecting to 172.23.98.19 with username:root password:couchbase ssh_key:
[2018-08-26 17:13:24,124] - [remote_util:266] INFO - Connected to 172.23.98.19
[2018-08-26 17:13:24,396] - [remote_util:3297] INFO - It could be other operating system. Go to check at other location
[2018-08-26 17:13:24,801] - [remote_util:749] INFO - 172.23.98.19 **** The version file /opt/couchbase/ VERSION.txt exists
[2018-08-26 17:13:24,802] - [basetestcase:489] INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ...
[2018-08-26 17:13:29,813] - [basetestcase:489] INFO - sleep for 10 secs. Wait for ns server is ready ...
[2018-08-26 17:13:39,813] - [upgrade_tests:200] INFO - record vbucket for the bucket default
[2018-08-26 17:13:40,042] - [rest_client:2026] INFO - Node versions in cluster [u'5.0.0-3519-enterprise', u'5.0.0-3519-enterprise', u'6.0.0-1564-enterprise']
[2018-08-26 17:13:40,081] - [task:431] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:13:40,081] - [task:446] INFO - adding node 172.23.98.19:8091 to cluster
[2018-08-26 17:13:40,081] - [rest_client:1244] INFO - adding remote node @172.23.98.19:8091 to this cluster @172.23.98.11:8091
[2018-08-26 17:13:42,009] - [rest_client:1462] INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.98.11', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.98.19,ns_1@172.23.98.214,ns_1@172.23.97.230,ns_1@172.23.98.11'}
[2018-08-26 17:13:42,015] - [rest_client:1467] INFO - rebalance operation started
[2018-08-26 17:13:42,015] - [task:497] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:13:42,058] - [rest_client:1615] INFO - rebalance percentage : 0.00 %
[2018-08-26 17:13:42,058] - [task:512] INFO - Rebalance - status: running, progress: 0.00%
[2018-08-26 17:13:52,075] - [task:497] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:13:52,138] - [rest_client:1615] INFO - rebalance percentage : 44.72 %
[2018-08-26 17:13:52,138] - [task:512] INFO - Rebalance - status: running, progress: 44.72%
[2018-08-26 17:14:02,157] - [task:497] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:14:02,204] - [rest_client:1615] INFO - rebalance percentage : 50.00 %
[2018-08-26 17:14:02,205] - [task:512] INFO - Rebalance - status: running, progress: 50.00%
[2018-08-26 17:14:12,230] - [task:497] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:14:12,273] - [rest_client:1615] INFO - rebalance percentage : 50.00 %
[2018-08-26 17:14:12,273] - [task:512] INFO - Rebalance - status: running, progress: 50.00%
[2018-08-26 17:14:22,289] - [task:497] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:14:22,328] - [rest_client:1615] INFO - rebalance percentage : 50.00 %
[2018-08-26 17:14:22,328] - [task:512] INFO - Rebalance - status: running, progress: 50.00%
[2018-08-26 17:14:32,345] - [task:497] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:14:32,389] - [rest_client:1615] INFO - rebalance percentage : 50.00 %
[2018-08-26 17:14:32,389] - [task:512] INFO - Rebalance - status: running, progress: 50.00%
[2018-08-26 17:14:42,407] - [task:497] INFO - This is swap rebalance and we will monitor vbuckets shuffling
[2018-08-26 17:14:42,449] - [rest_client:1598] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
[2018-08-26 17:14:42,459] - [rest_client:3134] INFO - Latest logs from UI on 172.23.98.11:
[2018-08-26 17:14:42,459] - [rest_client:3135] ERROR - {u'node': u'ns_1@172.23.98.214', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,fts,\n {linked_process_died,<20889.1674.0>,\n {no_connection,"fts-service_api"}}}', u'shortText': u'message', u'serverTime': u'2018-08-26T17:14:42.057Z', u'module': u'ns_orchestrator', u'tstamp': 1535328882057, u'type': u'critical'}
[2018-08-26 17:14:42,459] - [rest_client:3135] ERROR - {u'node': u'ns_1@172.23.98.19', u'code': 0, u'text': u"Service 'fts' exited with status 1. Restarting. Messages:\n2018-08-26T17:14:38.427-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:38.428-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:38.428-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:38.429-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:38.429-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:38.429-07:00 [INFO] audit: created new audit service\n2018-08-26T17:14:38.432-07:00 [INFO] main: ctl starting, dryRun: false, waitForMemberNodes: 30, verbose: 3, maxConcurrentPartitionMovesPerNode: 1\n2018-08-26T17:14:38.438-07:00 [INFO] ctl: run, PlannerGetPlanPIndexes err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0\n2018-08-26T17:14:38.438-07:00 [FATA] main: ctl.StartCtl, err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0 -- main.main() at main.go:227\n[goport(/opt/couchbase/bin/cbft)] 2018/08/26 17:14:38 child process exited with status 1\n", u'shortText': u'message', u'serverTime': u'2018-08-26T17:14:38.441Z', u'module': u'ns_log', u'tstamp': 1535328878441, u'type': u'info'}
[2018-08-26 17:14:42,459] - [rest_client:3135] ERROR - {u'node': u'ns_1@172.23.98.19', u'code': 0, u'text': u"Service 'fts' exited with status 1. Restarting. Messages:\n2018-08-26T17:14:33.327-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:33.327-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:33.328-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:33.328-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:33.328-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:33.329-07:00 [INFO] audit: created new audit service\n2018-08-26T17:14:33.335-07:00 [INFO] main: ctl starting, dryRun: false, waitForMemberNodes: 30, verbose: 3, maxConcurrentPartitionMovesPerNode: 1\n2018-08-26T17:14:33.340-07:00 [INFO] ctl: run, PlannerGetPlanPIndexes err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0\n2018-08-26T17:14:33.340-07:00 [FATA] main: ctl.StartCtl, err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0 -- main.main() at main.go:227\n[goport(/opt/couchbase/bin/cbft)] 2018/08/26 17:14:33 child process exited with status 1\n", u'shortText': u'message', u'serverTime': u'2018-08-26T17:14:33.343Z', u'module': u'ns_log', u'tstamp': 1535328873343, u'type': u'info'}
[2018-08-26 17:14:42,459] - [rest_client:3135] ERROR - {u'node': u'ns_1@172.23.98.19', u'code': 0, u'text': u"Service 'fts' exited with status 1. Restarting. Messages:\n2018-08-26T17:14:28.229-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:28.229-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:28.229-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:28.230-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:28.230-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:28.230-07:00 [INFO] audit: created new audit service\n2018-08-26T17:14:28.235-07:00 [INFO] main: ctl starting, dryRun: false, waitForMemberNodes: 30, verbose: 3, maxConcurrentPartitionMovesPerNode: 1\n2018-08-26T17:14:28.240-07:00 [INFO] ctl: run, PlannerGetPlanPIndexes err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0\n2018-08-26T17:14:28.240-07:00 [FATA] main: ctl.StartCtl, err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0 -- main.main() at main.go:227\n[goport(/opt/couchbase/bin/cbft)] 2018/08/26 17:14:28 child process exited with status 1\n", u'shortText': u'message', u'serverTime': u'2018-08-26T17:14:28.243Z', u'module': u'ns_log', u'tstamp': 1535328868243, u'type': u'info'}
[2018-08-26 17:14:42,459] - [rest_client:3135] ERROR - {u'node': u'ns_1@172.23.98.19', u'code': 0, u'text': u"Service 'fts' exited with status 1. Restarting. Messages:\n2018-08-26T17:14:23.131-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:23.132-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:23.132-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:23.133-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:23.133-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:23.133-07:00 [INFO] audit: created new audit service\n2018-08-26T17:14:23.136-07:00 [INFO] main: ctl starting, dryRun: false, waitForMemberNodes: 30, verbose: 3, maxConcurrentPartitionMovesPerNode: 1\n2018-08-26T17:14:23.141-07:00 [INFO] ctl: run, PlannerGetPlanPIndexes err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0\n2018-08-26T17:14:23.141-07:00 [FATA] main: ctl.StartCtl, err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0 -- main.main() at main.go:227\n[goport(/opt/couchbase/bin/cbft)] 2018/08/26 17:14:23 child process exited with status 1\n", u'shortText': u'message', u'serverTime': u'2018-08-26T17:14:23.144Z', u'module': u'ns_log', u'tstamp': 1535328863144, u'type': u'info'}
[2018-08-26 17:14:42,459] - [rest_client:3135] ERROR - {u'node': u'ns_1@172.23.98.19', u'code': 0, u'text': u"Service 'fts' exited with status 1. Restarting. Messages:\n2018-08-26T17:14:18.031-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:18.031-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:18.032-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:18.032-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:18.032-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:18.032-07:00 [INFO] audit: created new audit service\n2018-08-26T17:14:18.036-07:00 [INFO] main: ctl starting, dryRun: false, waitForMemberNodes: 30, verbose: 3, maxConcurrentPartitionMovesPerNode: 1\n2018-08-26T17:14:18.040-07:00 [INFO] ctl: run, PlannerGetPlanPIndexes err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0\n2018-08-26T17:14:18.040-07:00 [FATA] main: ctl.StartCtl, err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0 -- main.main() at main.go:227\n[goport(/opt/couchbase/bin/cbft)] 2018/08/26 17:14:18 child process exited with status 1\n", u'shortText': u'message', u'serverTime': u'2018-08-26T17:14:18.043Z', u'module': u'ns_log', u'tstamp': 1535328858043, u'type': u'info'}
[2018-08-26 17:14:42,460] - [rest_client:3135] ERROR - {u'node': u'ns_1@172.23.98.19', u'code': 0, u'text': u"Service 'fts' exited with status 1. Restarting. Messages:\n2018-08-26T17:14:12.933-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:12.933-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:12.934-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:12.934-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:12.934-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:12.934-07:00 [INFO] audit: created new audit service\n2018-08-26T17:14:12.937-07:00 [INFO] main: ctl starting, dryRun: false, waitForMemberNodes: 30, verbose: 3, maxConcurrentPartitionMovesPerNode: 1\n2018-08-26T17:14:12.942-07:00 [INFO] ctl: run, PlannerGetPlanPIndexes err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0\n2018-08-26T17:14:12.942-07:00 [FATA] main: ctl.StartCtl, err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0 -- main.main() at main.go:227\n[goport(/opt/couchbase/bin/cbft)] 2018/08/26 17:14:12 child process exited with status 1\n", u'shortText': u'message', u'serverTime': u'2018-08-26T17:14:12.944Z', u'module': u'ns_log', u'tstamp': 1535328852944, u'type': u'info'}
[2018-08-26 17:14:42,460] - [rest_client:3135] ERROR - {u'node': u'ns_1@172.23.98.19', u'code': 0, u'text': u"Service 'fts' exited with status 1. Restarting. Messages:\n2018-08-26T17:14:07.833-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:07.833-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:07.834-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:07.834-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:07.834-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:07.834-07:00 [INFO] audit: created new audit service\n2018-08-26T17:14:07.838-07:00 [INFO] main: ctl starting, dryRun: false, waitForMemberNodes: 30, verbose: 3, maxConcurrentPartitionMovesPerNode: 1\n2018-08-26T17:14:07.843-07:00 [INFO] ctl: run, PlannerGetPlanPIndexes err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0\n2018-08-26T17:14:07.843-07:00 [FATA] main: ctl.StartCtl, err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0 -- main.main() at main.go:227\n[goport(/opt/couchbase/bin/cbft)] 2018/08/26 17:14:07 child process exited with status 1\n", u'shortText': u'message', u'serverTime': u'2018-08-26T17:14:07.845Z', u'module': u'ns_log', u'tstamp': 1535328847845, u'type': u'info'}
[2018-08-26 17:14:42,460] - [rest_client:3135] ERROR - {u'node': u'ns_1@172.23.98.11', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.98.11\' for deletion', u'shortText': u'message', u'serverTime': u'2018-08-26T17:14:03.569Z', u'module': u'ns_memcached', u'tstamp': 1535328843569, u'type': u'info'}
[2018-08-26 17:14:42,460] - [rest_client:3135] ERROR - {u'node': u'ns_1@172.23.98.19', u'code': 0, u'text': u"Service 'fts' exited with status 1. Restarting. Messages:\n2018-08-26T17:14:02.736-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:02.737-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:02.737-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:02.737-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:02.738-07:00 [INFO] Using plain authentication for user <ud>@fts</ud> \n2018-08-26T17:14:02.738-07:00 [INFO] audit: created new audit service\n2018-08-26T17:14:02.740-07:00 [INFO] main: ctl starting, dryRun: false, waitForMemberNodes: 30, verbose: 3, maxConcurrentPartitionMovesPerNode: 1\n2018-08-26T17:14:02.745-07:00 [INFO] ctl: run, PlannerGetPlanPIndexes err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0\n2018-08-26T17:14:02.745-07:00 [FATA] main: ctl.StartCtl, err: planner: planPIndexesPrev.ImplVersion: 5.5.0 > version: 5.0.0 -- main.main() at main.go:227\n[goport(/opt/couchbase/bin/cbft)] 2018/08/26 17:14:02 child process exited with status 1\n", u'shortText': u'message', u'serverTime': u'2018-08-26T17:14:02.747Z', u'module': u'ns_log', u'tstamp': 1535328842747, u'type': u'info'}
[('/usr/lib64/python2.7/threading.py', 785, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 812, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 77, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 523, 'check', 'self.set_exception(ex)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
Sun Aug 26 17:14:42 2018
[('/usr/lib64/python2.7/threading.py', 785, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 812, '__bootstrap_inner', 'self.run()'), ('/usr/lib64/python2.7/threading.py', 765, 'run', 'self.__target(*self.__args, **self.__kwargs)'), ('pytests/upgrade/upgrade_tests.py', 816, 'online_upgrade', 'self.online_upgrade_swap_rebalance()'), ('pytests/upgrade/upgrade_tests.py', 868, 'online_upgrade_swap_rebalance', 'services=[servicesNodeOut])'), ('lib/couchbase_helper/cluster.py', 333, 'rebalance', 'return _task.result(timeout)'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')]
[2018-08-26 17:14:42,463] - [upgrade_tests:881] INFO - Rebalance Failed: {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
[2018-08-26 17:14:42,463] - [upgrade_tests:820] INFO - Rebalance Failed: {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment