Last active
February 10, 2017 16:34
-
-
Save lbragstad/7e83a983fd83f63074feb600910de2f8 to your computer and use it in GitHub Desktop.
Using OSA to test rolling upgrades
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| sudo apt-get update; sudo apt-get dist-upgrade | |
| reboot |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| sudo apt-get install tmux python-pip | |
| git clone https://github.com/openstack/openstack-ansible-os_keystone | |
| git fetch https://git.openstack.org/openstack/openstack-ansible-os_keystone refs/changes/25/431225/6 | |
| git checkout FETCH_HEAD | |
| sudo pip install --upgrade pip | |
| sudo pip install tox |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| TASK [os_tempest : Install tempest plugin requirements] ************************ | |
| Friday 10 February 2017 14:49:45 +0000 (0:00:00.408) 0:19:42.123 ******* | |
| FAILED - RETRYING: TASK: os_tempest : Install tempest plugin requirements (5 retries left). | |
| FAILED - RETRYING: TASK: os_tempest : Install tempest plugin requirements (4 retries left). | |
| FAILED - RETRYING: TASK: os_tempest : Install tempest plugin requirements (3 retries left). | |
| FAILED - RETRYING: TASK: os_tempest : Install tempest plugin requirements (2 retries left). | |
| FAILED - RETRYING: TASK: os_tempest : Install tempest plugin requirements (1 retries left). | |
| failed: [keystone1] (item={'item': {u'name': u'keystone', u'branch': u'master', u'repo': u'https://git.openstack.org/openstack/keystone'}, '_ansible_item_result': True, u'stat': {u'inode': 16151356, u'path': u'/opt/keystone_master/test-requirements.txt', u'xusr': False, | |
| u'pw_name': u'root', u'readable': True, u'exists': True, u'executable': False, u'isuid': False, u'isgid': False, u'rusr': True, u'wusr': True, u'woth': False, u'dev': 51713, u'rgrp': True, u'gr_name': u'root', u'md5': u'28e7d086b247cbb59267eb4430959303', u'isblk': Fals | |
| e, u'islnk': False, u'isreg': True, u'ischr': False, u'checksum': u'bee96183f986a706461ed1f0db8081802a8f3932', u'wgrp': False, u'mode': u'0644', u'xoth': False, u'isdir': False, u'writeable': True, u'isfifo': False, u'issock': False, u'atime': 1486738184.3609154, u'roth | |
| ': True, u'uid': 0, u'xgrp': False, u'nlink': 1, u'ctime': 1486738184.3609154, u'mtime': 1486738184.3609154, u'size': 1169, u'gid': 0}, '_ansible_parsed': True, '_ansible_no_log': False, 'invocation': {'module_name': u'stat', u'module_args': {u'get_checksum': True, u'ch | |
| ecksum_algorithm': u'sha1', u'get_md5': True, u'follow': False, u'path': u'/opt/keystone_master/test-requirements.txt', u'mime': False}}, u'changed': False}) => {"attempts": 5, "cmd": "/opt/tempest_master/bin/pip2 install -U --constraint /opt/developer-pip-constraints.t | |
| xt --constraint http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt --no-binary libvirt-python -r /opt/keystone_master/test-requirements.txt", "failed": true, "item": {"changed": false, "invocation": {"module_args": {"checksum_algorithm": "sha1", "follow": false, "get_checksum": true, "get_md5": true, "mime": false, "path": "/opt/keystone_master/test-requirements.txt"}, "module_name": "stat"}, "item": {"branch": "master", "name": "keystone", "repo": "https://git.openstack.org/openstack/keystone"}, "stat": {"atime": 1486738184.3609154, "checksum": "bee96183f986a706461ed1f0db8081802a8f3932", "ctime": 1486738184.3609154, "dev": 51713, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 16151356, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "md5": "28e7d086b247cbb59267eb4430959303", "mode": "0644", "mtime": 1486738184.3609154, "nlink": 1, "path": "/opt/keystone_master/test-requirements.txt", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 1169, "uid": 0, "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}}, "msg": "stdout: Collecting tempest from git+https://git.openstack.org/openstack/tempest@master#egg=tempest (from -c /opt/developer-pip-constraints.txt (line 1))\n Cloning https://git.openstack.org/openstack/tempest (to master) to ./pip-build-pHS1WN/tempest\nRequirement already up-to-date: PyYAML===3.12 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 21))\nCollecting Sphinx===1.3.6 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 27))\n Using cached Sphinx-1.3.6-py2.py3-none-any.whl\nCollecting WebTest===2.0.24 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 31))\n Using cached WebTest-2.0.24-py2.py3-none-any.whl\nCollecting alabaster===0.7.9 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 65))\n Using cached alabaster-0.7.9-py2.py3-none-any.whl\nCollecting bashate===0.5.1 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 77))\n Using cached bashate-0.5.1-py2-none-any.whl\nCollecting beautifulsoup4===4.5.3 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 78))\n Using cached beautifulsoup4-4.5.3-py2-none-any.whl\nRequirement already up-to-date: cliff===2.4.0 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 93))\nRequirement already up-to-date: cmd2===0.6.9 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 95))\nCollecting coverage===4.3.4 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 99))\n Using cached coverage-4.3.4-cp27-cp27mu-manylinux1_x86_64.whl\nRequirement already up-to-date: debtcollector===1.11.0 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 105))\nCollecting docutils===0.12 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 125))\n Using cached docutils-0.12.tar.gz\nRequirement already up-to-date: fixtures===3.0.0 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 137))\nCollecting freezegun===0.3.8 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 139))\n Using cached freezegun-0.3.8-py2.py3-none-any.whl\nRequirement already up-to-date: funcsigs===1.0.2 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 142))\nRequirement already up-to-date: jsonschema===2.5.1 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 173))\nCollecting lxml===3.7.2 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 188))\n Using cached lxml-3.7.2-cp27-cp27mu-manylinux1_x86_64.whl\nCollecting mock===2.0.0 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 192))\n Using cached mock-2.0.0-py2.py3-none-any.whl\nRequirement already up-to-date: netaddr===0.7.19 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 202))\nCollecting os-api-ref===1.2.0 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 222))\n Using cached os_api_ref-1.2.0-py2-none-any.whl\nRequirement already up-to-date: os-testr===0.8.0 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 230))\nRequirement already up-to-date: oslo.concurrency===3.18.0 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 236))\nRequirement already up-to-date: oslo.config===3.22.0 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 237))\nCollecting oslo.db===4.17.0 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 239))\n Using cached oslo.db-4.17.0-py2.py3-none-any.whl\nRequirement already up-to-date: oslo.i18n===3.12.0 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 240))\nRequirement already up-to-date: oslo.log===3.20.0 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 241))\nRequirement already up-to-date: oslo.serialization===2.16.0 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 248))\nRequirement already up-to-date: oslo.utils===3.22.0 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 250))\nCollecting oslosphinx===4.10.0 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 253))\n Using cached oslosphinx-4.10.0-py2.py3-none-any.whl\nCollecting oslotest===2.13.0 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 254))\n Using cached oslotest-2.13.0-py2.py3-none-any.whl\nRequirement already up-to-date: paramiko===2.1.1 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 258))\nRequirement already up-to-date: pbr===1.10.0 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 262))\nRequirement already up-to-date: prettytable===0.7.2 in /opt/tempest_master/lib/python2.7/site-packages (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 270))\nCollecting psycopg2===2.6.2 (from -c http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt (line 275))\n Using cached psycopg2-2.6.2.tar.gz\n Complete output from command python setup.py egg_info:\n running egg_info\n creating pip-egg-info/psycopg2.egg-info\n writing pip-egg-info/psycopg2.egg-info/PKG-INFO\n writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt\n writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt\n writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'\n Error: pg_config executable not found.\n \n Please add the directory containing pg_config to the PATH\n or specify the full executable path with the option:\n \n python setup.py build_ext --pg-config /path/to/pg_config build ...\n \n or with the pg_config option in 'setup.cfg'.\n \n ----------------------------------------\n\n:stderr: Ignoring dnspython3: markers 'python_version == \"3.4\"' don't match your environment\nIgnoring dnspython3: markers 'python_version == \"3.5\"' don't match your environment\nCommand \"python setup.py egg_info\" failed with error code 1 in /tmp/pip-build-pHS1WN/psycopg2/\n"} | |
| to retry, use: --limit @/home/ubuntu/openstack-ansible-os_keystone/tests/test-upgrade.retry | |
| PLAY RECAP ********************************************************************* | |
| infra1 : ok=166 changed=91 unreachable=0 failed=0 | |
| keystone1 : ok=265 changed=107 unreachable=0 failed=1 | |
| keystone2 : ok=183 changed=80 unreachable=0 failed=0 | |
| localhost : ok=114 changed=16 unreachable=0 failed=0 | |
| Friday 10 February 2017 14:51:42 +0000 (0:01:57.075) 0:21:39.199 ******* | |
| =============================================================================== | |
| os_previous_keystone : Install pip packages --------------------------- 132.49s | |
| os_tempest : Install tempest plugin requirements ---------------------- 117.08s | |
| os_keystone : Install pip packages ------------------------------------ 104.05s | |
| os_tempest : Install pip packages -------------------------------------- 39.94s | |
| galera_server : Confirm service connectivity --------------------------- 32.59s | |
| galera_server : Install galera packages -------------------------------- 24.63s | |
| os_previous_keystone : Install apt packages ---------------------------- 23.28s | |
| os_previous_keystone : Install required pip packages ------------------- 21.63s | |
| lxc_container_create : Create container -------------------------------- 21.53s | |
| lxc_hosts : Create base container -------------------------------------- 20.78s | |
| galera_server : Start galera cluster ----------------------------------- 15.13s | |
| os_tempest : Install requires pip packages ----------------------------- 14.16s | |
| lxc_hosts : Create lxc image ------------------------------------------- 14.06s | |
| os_tempest : Get tempest plugins from git ------------------------------ 13.83s | |
| galera_server : Install pip packages ----------------------------------- 12.13s | |
| os_keystone : Copy keystone config ------------------------------------- 11.72s | |
| os_keystone : Wait for keystone admin to come up ----------------------- 10.37s | |
| os_keystone : Wait for keystone service to come up --------------------- 10.37s | |
| os_previous_keystone : Wait for keystone service to come up ------------ 10.36s | |
| os_previous_keystone : Wait for keystone admin to come up -------------- 10.35s | |
| ERROR: InvocationError: '/bin/bash -c /home/ubuntu/openstack-ansible-os_keystone/tests/common/test-ansible-functional.sh' | |
| __________________________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________________________ | |
| ERROR: upgrade: commands failed |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| DEBUG MESSAGE RECAP ************************************************************ | |
| DEBUG: [Display upgrade benchmarking result] *********************************** | |
| /usr/local/lib/python2.7/dist-packages/locust/rpc/__init__.py:6: UserWarning: WARNING: Using pure Python socket RPC implementation instead of zmq. If running in distributed mode, this could cause a performance decrease. We recommend you to install the pyzmq python package when running in distributed mode. | |
| warnings.warn("WARNING: Using pure Python socket RPC implementation instead of zmq. If running in distributed mode, this could cause a performance decrease. We recommend you to install the pyzmq python package when running in distributed mode.") | |
| Name # reqs # fails Avg Min Max | Median req/s | |
| -------------------------------------------------------------------------------------------------------------------------------------------- | |
| GET /v3/auth/tokens 1263 6(0.47%) 212 106 3321 | 130 2.90 | |
| POST /v3/auth/tokens 1264 4(0.32%) 241 170 4964 | 190 2.90 | |
| -------------------------------------------------------------------------------------------------------------------------------------------- | |
| Total 2527 10(0.40%) 5.80 | |
| Percentage of the requests completed within given times | |
| Name # reqs 50% 66% 75% 80% 90% 95% 98% 99% 100% | |
| -------------------------------------------------------------------------------------------------------------------------------------------- | |
| GET /v3/auth/tokens 1263 130 300 310 320 340 370 400 420 3321 | |
| POST /v3/auth/tokens 1264 190 200 200 210 230 260 290 3200 4964 | |
| -------------------------------------------------------------------------------------------------------------------------------------------- | |
| Error report | |
| # occurences Error | |
| -------------------------------------------------------------------------------------------------------------------------------------------- | |
| 1 GET /v3/auth/tokens: "HTTPError(u'500 Server Error: Internal Server Error for url: http://10.1.0.1:5000/v3/auth/tokens',)" | |
| 2 GET /v3/auth/tokens: 'ConnectionError(ProtocolError(\'Connection aborted.\', BadStatusLine("\'\'",)),)' | |
| 4 POST /v3/auth/tokens: 'ConnectionError(ProtocolError(\'Connection aborted.\', BadStatusLine("\'\'",)),)' | |
| 3 GET /v3/auth/tokens: "HTTPError(u'401 Client Error: Unauthorized for url: http://10.1.0.1:5000/v3/auth/tokens',)" | |
| -------------------------------------------------------------------------------------------------------------------------------------------- | |
| Thursday 09 February 2017 19:59:39 +0000 (0:00:00.043) 0:21:43.877 ***** | |
| =============================================================================== | |
| os_previous_keystone : Install pip packages --------------------------- 131.27s | |
| os_keystone : Install pip packages ------------------------------------- 99.98s | |
| os_tempest : Install tempest plugin requirements ----------------------- 52.90s | |
| os_tempest : Install pip packages -------------------------------------- 38.05s | |
| galera_server : Confirm service connectivity --------------------------- 32.41s | |
| os_tempest : Install tempest plugins ----------------------------------- 28.07s | |
| galera_server : Install galera packages -------------------------------- 24.55s | |
| os_previous_keystone : Install apt packages ---------------------------- 22.71s | |
| lxc_container_create : Create container -------------------------------- 21.84s | |
| os_previous_keystone : Install required pip packages ------------------- 21.37s | |
| lxc_hosts : Create base container -------------------------------------- 20.72s | |
| os_tempest : Execute tempest tests ------------------------------------- 20.07s | |
| memcached_server : Install apt packages -------------------------------- 15.59s | |
| os_tempest : Install requires pip packages ----------------------------- 14.02s | |
| pip_install : Ensure apt cache is up to date --------------------------- 13.78s | |
| lxc_hosts : Create lxc image ------------------------------------------- 12.96s | |
| galera_server : Install pip packages ----------------------------------- 12.14s | |
| os_keystone : Copy keystone config ------------------------------------- 11.53s | |
| os_tempest : Get tempest plugins from git ------------------------------ 10.44s | |
| os_keystone : Wait for keystone service to come up --------------------- 10.39s | |
| __________________________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________________________ | |
| upgrade: commands succeeded | |
| congratulations :) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| tox -e upgrade |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment