Skip to content

Instantly share code, notes, and snippets.

@irvingpop

irvingpop/EC-rsync-upgrade-process.md

Last active August 29, 2015 14:01
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save irvingpop/e4c41a0717ea17753aca to your computer and use it in GitHub Desktop.
Save irvingpop/e4c41a0717ea17753aca to your computer and use it in GitHub Desktop.
Download ZIP
Enterprise Chef rsync migration+upgrade procedure
Raw
EC-rsync-upgrade-process.md

Customer Scenario

A customer has an OPC 1.4.x HA installation in Production. They want to test an in-place upgrade from OPC 1.4.x all the way through to EC11.1.6 using their current OPC Production data and config. This gives us a good chance to make corrections if we find that their data is too broken for the migrations to handle, and gives the customer experience in managing the upgrade in Production.

The sequence of events will broadly be these:

  • Install OPC 1.4.11 on target HA Test cluster
  • Restore data from Production OPC 1.4.11 instance LVM snapshot backup produced by our drbd-backups script
  • Test
  • Upgrade to EC 11.1.6
  • Test

The Process

  1. Assuming a clean target system, nothing in /etc/opscode, /opt/opscode or /var/opt/opscode, no processes running which match ps -ef |grep opscode

Create the backup on the production cluster

  1. Create data backup from Production source system using the /opt/opscode/bin/drbd-backups tool (NOTE: this tool may need to be modified depending on the amount of available space in the LVM VG)
  2. Mount the backup volume (ex: /var/opt/opscode/drbd-backups )

On the target system: Install and Verify OPC 1.4.11

  1. Install OPC 1.4.11 on the backends and frontends as described in the documentation and validate the system is working correctly
  2. Shutdown secondary backend keepalived with private-chef-ctl stop
  3. Ensure that all services are correctly running on the primary backend: private-chef-ctl ha-status
  4. Shutdown Private Chef on the bootstrap Primary target system with: private-chef-ctl stop
  5. Force the drbd device to be primary with:
    drbdadm primary pc0
  6. Mount the DRBD volume: mount -t ext4 /dev/drbd0 /var/opt/opscode/drbd/data
  7. Copy data into proper locations on the current bootstrap Primary target system:
rsync -avz -e ssh --exclude chef-server-running.json --exclude private-chef.rb root@prodbackend1:/etc/opscode/ /etc/opscode

rsync -avz -e ssh root@prodbackend1:/var/opt/opscode/drbd-backups/ /var/opt/opscode/drbd/data
  1. Start postgresql with: private-chef-ctl postgresql start
  2. Drop opscode_chef and bifrost databases: 
    /opt/opscode/embedded/bin/psql -U opscode-pgsql -d postgres --command='DROP DATABASE opscode_chef'

/opt/opscode/embedded/bin/psql -U opscode-pgsql -d postgres --command='DROP DATABASE bifrost'
  3. Uncompress the backup: gzip -d /var/opt/opscode/drbd/data/pgsql-backup.out.gz
  4. Restore postgresql backup: /opt/opscode/embedded/bin/psql -U "opscode-pgsql" -d postgres -f pgsql-backup.out
  5. private-chef-ctl reconfigure
  6. private-chef-ctl start
  7. Copy /etc/opscode to the frontends and secondary backend
  8. Reconfigure the frontends and TEST

On the target system: Upgrade to EC11

Follow the instructions from this document: http://docs.opscode.com/upgrade_server_ha_notes.html

TEST

  1. On each system run private-chef-ctl test
  2. Check DRBD status to be sure we are replicating with the secondary
  3. Test a selection of orgs and operations on those orgs: client lists, node lists, chef-client runs, group memberships, and any other important or desired tests.

Installation Directions for EC11x

http://docs.opscode.com/enterprise/install_server_be.html http://docs.opscode.com/enterprise/install_server_fe.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment