Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save virtualhops/b673efaa8d3cd9dc9c060e5bf159ce5a to your computer and use it in GitHub Desktop.
Save virtualhops/b673efaa8d3cd9dc9c060e5bf159ce5a to your computer and use it in GitHub Desktop.
Contrail Upgrade procedure – Minimal downtime
WARNING: This procedure only works for upgrades between Contrail versions PRIOR TO 3.0. Starting with 3.0, Cassandra version changes and this involves more work.
NOTE: This procedure is *UNSUPPORTED* because fab adds much more error-checking logic than what you can see here. Use this procedure at your own risk.
This document is inspired on another one ( https://gist.github.com/dsundarraj/9dde07f28208737c7729 ), but it breaks the fab steps into commands run on each of the nodes separately.
Upgrade procedure to minimize downtime in the production network. Upgrade controller and compute separately. The procedure is based on fab upgrade_contrail.
1. Workaround for the bug (https://bugs.launchpad.net/juniperopenstack/+bug/1506634 )
Create policy to deny stop/start of services during upgrade.
On all configuration, analytics nodes:
echo "#! /bin/sh" > /usr/sbin/policy-rc.d
echo "exit 101" >> /usr/sbin/policy-rc.d
chmod 755 /usr/sbin/policy-rc.d
2. Transfer the image to /root of all servers (control, configuration, analytics, database, compute).
Then install contrail-install-packages and create the repo:
dpkg -i /root/contrail-install-packages_<Release>-<Build>~<OpenStack>_all.deb
/opt/contrail/contrail_packages/setup.sh
You may want to use a manual method to extract the necessary packages and create the repo in a customized manner for each server.
3. Stop the database clients and the message broker:
On all the configuration nodes:
service supervisor-config stop
service neutron-server stop
On all the Contrail rabbitmq nodes:
service supervisor-support-service stop
On all the analytics nodes:
service supervisor-analytics stop
4. Upgrade the database nodes.
On all the database nodes:
ln -sf /bin/true /sbin/chkconfig
mkdir /var/tmp/contrail-upgradesave
mkdir /var/tmp/contrail-upgradesave/etc
cp -rfp /etc/contrail /var/tmp/contrail-upgradesave/etc/
DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-openstack-database
cp -rfp /var/tmp/contrail-upgradesave/etc/contrail/contrail-database-nodemgr.conf /etc/contrail/contrail-database-nodemgr.conf
service zookeeper restart
service supervisor-database restart
5. Upgrade the config nodes.
On all the config nodes:
ln -sf /bin/true /sbin/chkconfig
mkdir /var/tmp/contrail-upgradesave
mkdir /var/tmp/contrail-upgradesave/etc
mkdir /var/tmp/contrail-upgradesave/etc/init.d
mkdir /var/tmp/contrail-upgradesave/etc/sudoers.d
cp -rfp /etc/contrail /var/tmp/contrail-upgradesave/etc/
cp -rfp /etc/ifmap-server /var/tmp/contrail-upgradesave/etc/
cp -rfp /etc/neutron /var/tmp/contrail-upgradesave/etc/
cp -rfp /etc/init.d/contrail-api /var/tmp/contrail-upgradesave/etc/init.d/
cp -rfp /etc/init.d/contrail-discovery /var/tmp/contrail-upgradesave/etc/init.d/
cp -rfp /etc/sudoers.d/contrail_sudoers /var/tmp/contrail-upgradesave/etc/sudoers.d/
DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-openstack-config
cp -rfp /var/tmp/contrail-upgradesave/etc/contrail /etc/
cp -rfp /var/tmp/contrail-upgradesave/etc/ifmap-server /etc/
cp -rfp /var/tmp/contrail-upgradesave/etc/neutron /etc/
cp -rfp /var/tmp/contrail-upgradesave/etc/init.d/contrail-api /etc/init.d/
cp -rfp /var/tmp/contrail-upgradesave/etc/init.d/contrail-discovery /etc/init.d/
cp -rfp /var/tmp/contrail-upgradesave/etc/sudoers.d/contrail_sudoers /etc/sudoers.d/
On all the Contrail rabbitmq nodes:
service supervisor-support-service restart
On all the configuration nodes:
service supervisor-config restart
service neutron-server restart
6. Upgrade the analytics nodes.
On all the analytics nodes:
mkdir /var/tmp/contrail-upgradesave
mkdir /var/tmp/contrail-upgradesave/etc
cp -rfp /etc/contrail /var/tmp/contrail-upgradesave/etc/
cp -rfp /etc/redis /var/tmp/contrail-upgradesave/etc/
DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-openstack-analytics
cp -rfp /var/tmp/contrail-upgradesave/etc/contrail /etc/
sudo service redis-server restart
service supervisor-analytics restart
7. Upgrade the control nodes one by one, so there is always at least one BGP session up to the gateway and XMPP sessions to the compute nodes.
On each control node:
mkdir /var/tmp/contrail-upgradesave
mkdir /var/tmp/contrail-upgradesave/etc
cp -rfp /etc/contrail /var/tmp/contrail-upgradesave/etc/
DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-openstack-control
cp -rfp /var/tmp/contrail-upgradesave/etc/contrail /etc/
service supervisor-control restart
The BGP and XMPP sessions on this control node flaps.
8. Upgrade the webui nodes.
On all the webui nodes:
DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-openstack-webui
cp /var/tmp/contrail-upgradesave/etc/contrail/config.global.js /etc/contrail/
cp /var/tmp/contrail-upgradesave/etc/contrail/contrail-webui-userauth.js /etc/contrail/
service redis-server restart
service supervisor-webui restart
9. Upgrade the compute nodes.
On all the compute nodes:
mkdir /var/tmp/contrail-upgradesave
mkdir /var/tmp/contrail-upgradesave/etc
cp -rfp /etc/nova /var/tmp/contrail-upgradesave/etc/
cp -rfp /etc/contrail /var/tmp/contrail-upgradesave/etc/
cp -rfp /etc/libvirt /var/tmp/contrail-upgradesave/etc/
ln -sf /bin/true /sbin/chkconfig
DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-vrouter-3.13.0-40-generic contrail-openstack-vrouter
cp -rfp /var/tmp/contrail-upgradesave/etc/contrail /etc/
cp -rfp /var/tmp/contrail-upgradesave/etc/nova /etc/
cp -rfp /var/tmp/contrail-upgradesave/etc/libvirt /etc/
service supervisor-vrouter stop; modprobe -r vrouter || rmmod vrouter; modprobe vrouter; service supervisor-vrouter start
service nova-compute restart
10. Remove workaround for the bug (https://bugs.launchpad.net/juniperopenstack/+bug/1506634 )
On all configuration, analytics nodes:
rm /usr/sbin/policy-rc.d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment