Skip to content

Instantly share code, notes, and snippets.

@nrb

nrb/01-description.md Secret

Last active August 29, 2015 14:26
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nrb/d6142c104677c09683f1 to your computer and use it in GitHub Desktop.
Save nrb/d6142c104677c09683f1 to your computer and use it in GitHub Desktop.
Download ZIP
Repo failures
Raw
01-description.md

During the upgrade process, the utility containers begin failing to install new packages. I see 2 problems

  1. The utility containers are looking at the internal LB VIP, which is not yet configured to handle the repo, since the F5 config script wasn't run yet; it should be using an IP to one of the containers directly, at least for install. (file 04)

  2. The repo containers appear to be missing their contents. See file 03. Neither infra01 nor infra02's repo container has a populated repository.

Re-running setup-infrastructure did not change either of these.

Trying openstack-ansible repo-build.yml resulted in the failure seen in file 05.

Raw
02-ansible-failure.bash
TASK: [galera_client | Install pip packages] **********************************
failed: [578127-infra02_utility_container-810372fe] => (item=MySQL-python) => {"attempts": 5, "cmd": "/usr/local/bin/pip install MySQL-python", "failed": true, "item": "MySQL-pyt
hon"}
msg: Task failed as maximum retries was encountered
failed: [578128-infra03_utility_container-b43715a5] => (item=MySQL-python) => {"attempts": 5, "cmd": "/usr/local/bin/pip install MySQL-python", "failed": true, "item": "MySQL-pyt
hon"}
msg: Task failed as maximum retries was encountered
failed: [578126-infra01_utility_container-5dae2dab] => (item=MySQL-python) => {"attempts": 5, "cmd": "/usr/local/bin/pip install MySQL-python", "failed": true, "item": "MySQL-pyt
hon"}
msg: Task failed as maximum retries was encountered
ok: [578127-infra02_utility_container-810372fe] => (item=python-memcached)
ok: [578128-infra03_utility_container-b43715a5] => (item=python-memcached)
ok: [578127-infra02_utility_container-810372fe] => (item=pycrypto)
ok: [578128-infra03_utility_container-b43715a5] => (item=pycrypto)
ok: [578126-infra01_utility_container-5dae2dab] => (item=python-memcached)
ok: [578126-infra01_utility_container-5dae2dab] => (item=pycrypto)
FATAL: all hosts have already failed -- aborting
PLAY RECAP ********************************************************************
to retry, use: --limit @/root/setup-infrastructure.retry
578126-infra01_galera_container-d95e5cd7 : ok=55 changed=25 unreachable=0 failed=0
578126-infra01_memcached_container-e4a1cd65 : ok=7 changed=4 unreachable=0 failed=0
578126-infra01_rabbit_mq_container-f8397bfd : ok=43 changed=24 unreachable=0 failed=0
578126-infra01_repo_container-56ab6b39 : ok=44 changed=30 unreachable=0 failed=0
578126-infra01_utility_container-5dae2dab : ok=16 changed=8 unreachable=0 failed=1
578127-infra02_galera_container-ed65ed97 : ok=55 changed=24 unreachable=0 failed=0
578127-infra02_memcached_container-17292b85 : ok=7 changed=4 unreachable=0 failed=0
578127-infra02_rabbit_mq_container-49525ceb : ok=42 changed=22 unreachable=0 failed=0
578127-infra02_repo_container-5c60a673 : ok=44 changed=30 unreachable=0 failed=0
578127-infra02_utility_container-810372fe : ok=16 changed=8 unreachable=0 failed=1
578128-infra03_galera_container-010493c9 : ok=55 changed=24 unreachable=0 failed=0
578128-infra03_memcached_container-8abbe982 : ok=7 changed=4 unreachable=0 failed=0
578128-infra03_rabbit_mq_container-4c96d7b0 : ok=48 changed=33 unreachable=0 failed=0
578128-infra03_repo_container-44b5dac7 : ok=46 changed=33 unreachable=0 failed=0
578128-infra03_utility_container-b43715a5 : ok=16 changed=8 unreachable=0 failed=1
ran -e 'rabbitmq_upgrade=true' setup-infrastructure.yml
******************** FAILURE ********************
The upgrade script has failed please rerun the following task to continue
Failed on task -e 'rabbitmq_upgrade=true' setup-infrastructure.yml
Do NOT rerun the upgrade script!
Please execute the remaining tasks:
-e 'rabbitmq_upgrade=true' setup-infrastructure.yml
setup-openstack.yml
******************** FAILURE ********************
root@578126-infra01:/opt/rpc-openstack#
Raw
03-repo-contents.bash
# infra01's repo appears to be empty
root@578126-infra01:/opt/rpc-openstack# lxc-attach -n 578126-infra01_repo_container-56ab6b39
root@578126-infra01_repo_container-56ab6b39:/# cd /var/www
root@578126-infra01_repo_container-56ab6b39:/var/www# ls
repo
root@578126-infra01_repo_container-56ab6b39:/var/www# cd repo/os-releases/
root@578126-infra01_repo_container-56ab6b39:/var/www/repo/os-releases# ls
11.1.0
root@578126-infra01_repo_container-56ab6b39:/var/www/repo/os-releases# cd 11.1.0/
root@578126-infra01_repo_container-56ab6b39:/var/www/repo/os-releases/11.1.0# ls
root@578126-infra01_repo_container-56ab6b39:/var/www/repo/os-releases/11.1.0# ls
root@578126-infra01_repo_container-56ab6b39:/var/www/repo/os-releases/11.1.0# ls -al
total 8
drwxr-sr-x 2 nginx www-data 4096 Aug 5 23:02 .
drwxr-sr-x 3 nginx www-data 4096 Aug 5 23:02 ..
root@578126-infra01_repo_container-56ab6b39:/var/www/repo/os-releases/11.1.0# exit
# infra02's repo also appears to be empty
root@578127-infra02_repo_container-5c60a673:~# cd /var/www/repo/
links/ openstackgit/ os-releases/ pools/ python_packages/ reports/
root@578127-infra02_repo_container-5c60a673:~# cd /var/www/repo/os-releases/11.1.0/
root@578127-infra02_repo_container-5c60a673:/var/www/repo/os-releases/11.1.0# ls
root@578127-infra02_repo_container-5c60a673:/var/www/repo/os-releases/11.1.0#
# nothing appears to be in the directory except for the top-level directories, which are themselves empty
root@578127-infra02_repo_container-5c60a673:/var/www/repo/os-releases/11.1.0# cd /var/www/repo
root@578127-infra02_repo_container-5c60a673:/var/www/repo# ls -al
total 32
drwxr-sr-x 8 nginx www-data 4096 Aug 5 23:02 .
drwxr-sr-x 5 nginx www-data 4096 Aug 5 23:02 ..
drwxr-sr-x 2 nginx www-data 4096 Aug 5 23:02 links
drwxr-sr-x 2 nginx www-data 4096 Aug 5 23:02 openstackgit
drwxr-sr-x 3 nginx www-data 4096 Aug 5 23:02 os-releases
drwxr-sr-x 2 nginx www-data 4096 Aug 5 23:02 pools
drwxr-sr-x 3 nginx www-data 4096 Aug 5 23:02 python_packages
drwxr-sr-x 2 nginx www-data 4096 Aug 5 23:02 reports
root@578127-infra02_repo_container-5c60a673:/var/www/repo# cd python_packages/
root@578127-infra02_repo_container-5c60a673:/var/www/repo/python_packages# ls
pools
root@578127-infra02_repo_container-5c60a673:/var/www/repo/python_packages# cd pools/
root@578127-infra02_repo_container-5c60a673:/var/www/repo/python_packages/pools# ls
root@578127-infra02_repo_container-5c60a673:/var/www/repo/python_packages/pools# cd ..
root@578127-infra02_repo_container-5c60a673:/var/www/repo/python_packages# cd .
root@578127-infra02_repo_container-5c60a673:/var/www/repo/python_packages# ls
pools
root@578127-infra02_repo_container-5c60a673:/var/www/repo/python_packages# cd ..
root@578127-infra02_repo_container-5c60a673:/var/www/repo# ls
links openstackgit os-releases pools python_packages reports
root@578127-infra02_repo_container-5c60a673:/var/www/repo# cd reports/
root@578127-infra02_repo_container-5c60a673:/var/www/repo/reports# ls
root@578127-infra02_repo_container-5c60a673:/var/www/repo/reports# cd ..
root@578127-infra02_repo_container-5c60a673:/var/www/repo# cd openstackgit/
root@578127-infra02_repo_container-5c60a673:/var/www/repo/openstackgit# ls
root@578127-infra02_repo_container-5c60a673:/var/www/repo/openstackgit# cd ..
root@578127-infra02_repo_container-5c60a673:/var/www/repo# ls
links openstackgit os-releases pools python_packages reports
root@578127-infra02_repo_container-5c60a673:/var/www/repo# cd links/
root@578127-infra02_repo_container-5c60a673:/var/www/repo/links# ls
root@578127-infra02_repo_container-5c60a673:/var/www/repo/links#
Raw
05-repo-build-error.bash
TASK: [Build all known python packages requirements] **************************
failed: [578128-infra03_repo_container-44b5dac7] => {"changed": true, "cmd": "yaprt --quiet build-wheels --report-file /var/www/repo/reports/release-11.1.0-report.json --storage-
pool \"/var/www/repo/pools\" --link-dir \"/var/www/repo/os-releases/11.1.0\" --pip-extra-link-dirs \"/var/www/repo/links\" --pip-index \"http://rpc-repo.rackspace.com/pools\" --p
ip-extra-index \"https://pypi.python.org/simple/\" --pip-bulk-operation --build-output \"/tmp/openstack-wheel-output\" --build-dir \"/tmp/openstack-builder\" --build-requirements
--force-clean", "delta": "0:03:12.302470", "end": "2015-08-06 01:19:51.653304", "rc": 1, "start": "2015-08-06 01:16:39.350834", "warnings": []}
stderr: /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This pre
vents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#
insecureplatformwarning.
InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents ur
llib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecure
platformwarning.
InsecurePlatformWarning
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 223, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/dist-packages/pip/commands/wheel.py", line 194, in run
if not wb.build():
File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 705, in build
self.requirement_set.prepare_files(self.finder)
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 334, in prepare_files
functools.partial(self._prepare_file, finder))
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 321, in _walk_req_to_install
more_reqs = handler(req_to_install)
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 491, in _prepare_file
session=self.session)
File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 814, in unpack_url
unpack_file_url(link, location, download_dir)
File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 723, in unpack_file_url
unpack_file(from_path, location, content_type, link)
File "/usr/local/lib/python2.7/dist-packages/pip/utils/__init__.py", line 644, in unpack_file
flatten=not filename.endswith('.whl')
File "/usr/local/lib/python2.7/dist-packages/pip/utils/__init__.py", line 528, in unzip_file
zipfp = open(filename, 'rb')
IOError: [Errno 2] No such file or directory: u'/var/www/repo/links/python_keystoneclient-2012.2-py2-none-any.whl'
FATAL: all hosts have already failed -- aborting
PLAY RECAP ********************************************************************
to retry, use: --limit @/root/repo-build.retry
578128-infra03_repo_container-44b5dac7 : ok=1 changed=1 unreachable=0 failed=1
root@578126-infra01:/opt/rpc-openstack/os-ansible-deployment/playbooks#
Raw
06-rebuilding-repo-server.sh
TASK: [repo_server | Install apt packages] ************************************
ok: [578127-infra02_repo_container-5c60a673] => (item=aptitude,bridge-utils,build-essential,cgroup-lite,cmake,dmeventd,fcgiwrap,git-core,iptables,lsyncd,lynx,nginx-extras,rsync,sshpass,tmux,liberasurecode1,liberasurecode-dev,libldap2-dev,libsasl2-dev,libssl-dev,libxslt1.1,libpq-dev,libffi-dev,libsqlite3-dev,libxml2-dev,libxslt1-dev,libkmod-dev,libkrb5-dev,libkmod2,libvirt-dev,libmysqlclient-dev,lxc-dev,lvm2,python-dev,python-software-properties,sqlite3,swig,vlan,uuid-dev)
ok: [578128-infra03_repo_container-44b5dac7] => (item=aptitude,bridge-utils,build-essential,cgroup-lite,cmake,dmeventd,fcgiwrap,git-core,iptables,lsyncd,lynx,nginx-extras,rsync,sshpass,tmux,liberasurecode1,liberasurecode-dev,libldap2-dev,libsasl2-dev,libssl-dev,libxslt1.1,libpq-dev,libffi-dev,libsqlite3-dev,libxml2-dev,libxslt1-dev,libkmod-dev,libkrb5-dev,libkmod2,libvirt-dev,libmysqlclient-dev,lxc-dev,lvm2,python-dev,python-software-properties,sqlite3,swig,vlan,uuid-dev)
failed: [578126-infra01_repo_container-56ab6b39] => (item=aptitude,bridge-utils,build-essential,cgroup-lite,cmake,dmeventd,fcgiwrap,git-core,iptables,lsyncd,lynx,nginx-extras,rsync,sshpass,tmux,liberasurecode1,liberasurecode-dev,libldap2-dev,libsasl2-dev,libssl-dev,libxslt1.1,libpq-dev,libffi-dev,libsqlite3-dev,libxml2-dev,libxslt1-dev,libkmod-dev,libkrb5-dev,libkmod2,libvirt-dev,libmysqlclient-dev,lxc-dev,lvm2,python-dev,python-software-properties,sqlite3,swig,vlan,uuid-dev) => {"attempts": 5, "failed": true, "item": "aptitude,bridge-utils,build-essential,cgroup-lite,cmake,dmeventd,fcgiwrap,git-core,iptables,lsyncd,lynx,nginx-extras,rsync,sshpass,tmux,liberasurecode1,liberasurecode-dev,libldap2-dev,libsasl2-dev,libssl-dev,libxslt1.1,libpq-dev,libffi-dev,libsqlite3-dev,libxml2-dev,libxslt1-dev,libkmod-dev,libkrb5-dev,libkmod2,libvirt-dev,libmysqlclient-dev,lxc-dev,lvm2,python-dev,python-software-properties,sqlite3,swig,vlan,uuid-dev"}
stderr: E: Unable to correct problems, you have held broken packages.
stdout: Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
libmysqlclient-dev : Depends: libmysqlclient18 (= 5.5.44-0ubuntu0.14.04.1) but 5.5.44+maria-1~trusty is to be installed
msg: Task failed as maximum retries was encountered
FATAL: all hosts have already failed -- aborting
PLAY RECAP ********************************************************************
to retry, use: --limit @/root/setup-infrastructure.retry
578126-infra01_memcached_container-e4a1cd65 : ok=6 changed=1 unreachable=0 failed=0
578126-infra01_repo_container-56ab6b39 : ok=17 changed=2 unreachable=0 failed=1
578127-infra02_memcached_container-17292b85 : ok=6 changed=1 unreachable=0 failed=0
578127-infra02_repo_container-5c60a673 : ok=18 changed=2 unreachable=0 failed=0
578128-infra03_memcached_container-8abbe982 : ok=6 changed=1 unreachable=0 failed=0
578128-infra03_repo_container-44b5dac7 : ok=18 changed=2 unreachable=0 failed=0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment