Created
March 15, 2016 18:41
-
-
Save dsundarraj/673619eb69cc436edec4 to your computer and use it in GitHub Desktop.
Contrail backup and restore
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Contrail database backup/restore procedure | |
| This procedure can be used to backup contrail database (except the ContrailAnalytics keyspace) during upgrade from R2.20#64 to R2.22.1#5. Incase of database corruption during the upgrade process, the backed up database can be restored on the cluster. The below procedure can be used to backup/restore contrail database except the ContrailAnalytics keyspace. | |
| 1.1 Dropping ContrailAnalytics keyspace before upgrade | |
| If the user prefers to drop the ContrailAnalytics data before upgrading the cluster, then the following can be done: | |
| From FAB_NODE, execute “fab stop_collector” | |
| On one of the database node, execute: (no need to execute this on all databse nodes) | |
| cqlsh <IP> 9160 | |
| cqlsh> drop keyspace “ContrailAnalytics”; | |
| In fab node, execute: | |
| FAB_NODE# fab -R database – "rm –rf /var/lib/cassandra/data/ContrailAnalytics" | |
| 1.2 Backup Contrail Database | |
| TO Backup the database except ContrailAnalytics (Refer to section “Backup Contrail Database”) | |
| The following nomenclature is used in this procedure: | |
| FAB_NODE : The node from where the fabric commands are run. | |
| BACKUP_SERVER : Server where the backed up data is stored | |
| DATABASE1, DATABASE2, DATABASE3 : Contrail database node 1,2,3. | |
| 1.2.1 Backup Zookeeper data | |
| FAB_NODE# fab stop_cfgm | |
| On BACKUP_SERVER create: | |
| mkdir -p <backup_path>/db1/var/lib/zookeeper/ | |
| mkdir -p <backup_path>/db2/var/lib/zookeeper/ | |
| mkdir -p <backup_path>/db3/var/lib/zookeeper/ | |
| DATABASE1# scp -r /var/lib/zookeeper/* root@<BACKUP_SERVER>:<backup_path>/db1/var/lib/zookeeper/ | |
| DATABASE2# scp -r /var/lib/zookeeper/* root@<BACKUP_SERVER>:<backup_path>/db2/var/lib/zookeeper/ | |
| DATABASE3# scp -r /var/lib/zookeeper/* root@<BACKUP_SERVER>:<backup_path>/db3/var/lib/zookeeper/ | |
| 1.2.2 Create snapshot of the database: | |
| Create database snapshot on all the database nodes using the following commands. | |
| FAB_NODE# fab -R database -- "nodetool -h localhost -p 7199 snapshot config_db_uuid" | |
| FAB_NODE# fab -R database -- "nodetool -h localhost -p 7199 snapshot to_bgp_keyspace" | |
| FAB_NODE# fab -R database -- "nodetool -h localhost -p 7199 snapshot svc_monitor_keyspace" | |
| 1.2.3 Copy the snapshots to the Backup Server | |
| 1.2.3.1 Copy config_db_uuid snapshot to the backup server | |
| DATABASE1# cd /var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/config_db_uuid/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done | |
| On BACKUP_SERVER create: | |
| mkdir -p <backup_path>/db1/<of all the directories listed during above step> | |
| DATABASE1# cd /var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db1/var/lib/cassandra/data/config_db_uuid/$snapshot_dir/; fi; done | |
| DATABASE2# cd /var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/config_db_uuid/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done | |
| On BACKUP_SERVER create: | |
| mkdir -p <backup_path>/db2/<of all the directories listed during above step> | |
| DATABASE2# cd /var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db2/var/lib/cassandra/data/config_db_uuid/$snapshot_dir/; fi; done | |
| DATABASE3# cd /var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/config_db_uuid/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done | |
| On BACKUP_SERVER create: | |
| mkdir -p <backup_path>/db3/<of all the directories listed during above step> | |
| DATABASE3# cd /var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db3/var/lib/cassandra/data/config_db_uuid/$snapshot_dir/; fi; done | |
| 1.2.3.2 Copy to_bgp_keyspace snapshot to the backup server | |
| DATABASE1# cd /var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/to_bgp_keyspace/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done | |
| On BACKUP_SERVER create: | |
| mkdir -p <backup_path>/db1/<of all the directories listed during above step> | |
| DATABASE1# cd /var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db1/var/lib/cassandra/data/to_bgp_keyspace/$snapshot_dir/; fi; done | |
| DATABASE2# cd /var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/to_bgp_keyspace/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done | |
| On BACKUP_SERVER create: | |
| mkdir -p <backup_path>/db2/<of all the directories listed during above step> | |
| DATABASE2# cd /var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db2/var/lib/cassandra/data/to_bgp_keyspace/$snapshot_dir/; fi; done | |
| DATABASE3# cd /var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/to_bgp_keyspace/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done | |
| On BACKUP_SERVER create: | |
| mkdir -p <backup_path>/db3/<of all the directories listed during above step> | |
| DATABASE3# cd /var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db3/var/lib/cassandra/data/to_bgp_keyspace/$snapshot_dir/; fi; done | |
| 1.2.3.3 Copy svc_monitor_keyspace snapshot to the backup server | |
| DATABASE1# cd /var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/svc_monitor_keyspace/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done | |
| On BACKUP_SERVER create: | |
| mkdir -p <backup_path>/db1/<of all the directories listed during above step> | |
| DATABASE1# cd /var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db1/var/lib/cassandra/data/svc_monitor_keyspace/$snapshot_dir/; fi; done | |
| DATABASE2# cd /var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/svc_monitor_keyspace/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done | |
| On BACKUP_SERVER create: | |
| mkdir -p <backup_path>/db2/<of all the directories listed during above step> | |
| DATABASE2# cd /var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db2/var/lib/cassandra/data/svc_monitor_keyspace/$snapshot_dir/; fi; done | |
| DATABASE3# cd /var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/svc_monitor_keyspace/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done | |
| On BACKUP_SERVER create: | |
| mkdir -p <backup_path>/db3/<of all the directories listed during above step> | |
| DATABASE3# cd /var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db3/var/lib/cassandra/data/svc_monitor_keyspace/$snapshot_dir/; fi; done | |
| 1.2.3.4 Copy the database initial tokens | |
| On BACKUP_SERVER create: | |
| mkdir -p <backup_path>/db1/initial_tokens/ | |
| mkdir -p <backup_path>/db2/initial_tokens/ | |
| mkdir -p <backup_path>/db3/initial_tokens/ | |
| DATABASE1: nodetool ring | grep <DATABASE1_Internal_IP> | awk '{print $NF ","}' | xargs > /tmp/initial_tokens | |
| DATABASE1: scp /tmp/initial_tokens root@<BACKUP_SERVER>:<backup_path>/db1/initial_tokens/ | |
| DATABASE2: nodetool ring | grep <DATABASE2_Internal_IP> | awk '{print $NF ","}' | xargs > /tmp/initial_tokens | |
| DATABASE2: scp /tmp/initial_tokens root@<BACKUP_SERVER>:<backup_path>/db2/initial_tokens/ | |
| DATABASE3: nodetool ring | grep <DATABASE3_Internal_IP> | awk '{print $NF ","}' | xargs > /tmp/initial_tokens | |
| DATABASE3: scp /tmp/initial_tokens root@<BACKUP_SERVER>:<backup_path>/db3/initial_tokens/ | |
| 1.3 Restore Contrail Database | |
| fab stop_cfgm | |
| fab stop_database | |
| 1.3.1 Restore Zookeeper data | |
| Copy the previously backed up corresponding /var/lib/zookeeper/* files from the BACKUP_SERVER | |
| FAB_NODE# fab -R database -- "chown -R zookeeper:zookeeper /var/lib/zookeeper/" | |
| ON DATABASE1,2,3 | |
| Delete the following | |
| /var/lib/cassandra/data/config_db_uuid/* | |
| /var/lib/cassandra/data/to_bgp_keyspace/* | |
| /var/lib/cassandra/data/svc_monitor_keyspace/* | |
| 1.3.2 Restore Cassandra data | |
| 1.3.2.1 Restore config_db_uuid keyspace | |
| On BACKUP_SERVER | |
| cd <backup_path>/db1/var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE1> "mkdir -p /var/lib/cassandra/data/config_db_uuid/$dir"; scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE1>:/var/lib/cassandra/data/config_db_uuid/$dir/; fi; done | |
| cd <backup_path>/db2/var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE2> "mkdir -p /var/lib/cassandra/data/config_db_uuid/$dir"; scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE2>:/var/lib/cassandra/data/config_db_uuid/$dir/; fi; done | |
| cd <backup_path>/db3/var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE3> "mkdir -p /var/lib/cassandra/data/config_db_uuid/$dir"; scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE3>:/var/lib/cassandra/data/config_db_uuid/$dir/; fi; done | |
| 1.3.2.2 Restore to_bgp_keyspace keyspace | |
| On BACKUP_SERVER | |
| cd <backup_path>/db1/var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE1> "mkdir -p /var/lib/cassandra/data/to_bgp_keyspace/$dir"; scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE1>:/var/lib/cassandra/data/to_bgp_keyspace/$dir/; fi; done | |
| cd <backup_path>/db2/var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE2> "mkdir -p /var/lib/cassandra/data/to_bgp_keyspace/$dir"; scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE2>:/var/lib/cassandra/data/to_bgp_keyspace/$dir/; fi; done | |
| cd <backup_path>/db3/var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE3> "mkdir -p /var/lib/cassandra/data/to_bgp_keyspace/$dir"; scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE3>:/var/lib/cassandra/data/to_bgp_keyspace/$dir/; fi; done | |
| 1.3.2.3 Restore svc_monitor_keyspace keyspace | |
| On BACKUP_SERVER | |
| cd <backup_path>/db1/var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE1> "mkdir -p /var/lib/cassandra/data/svc_monitor_keyspace/$dir"; scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE1>:/var/lib/cassandra/data/svc_monitor_keyspace/$dir/; fi; done | |
| cd <backup_path>/db2/var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE2> "mkdir -p /var/lib/cassandra/data/svc_monitor_keyspace/$dir"; scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE2>:/var/lib/cassandra/data/svc_monitor_keyspace/$dir/; fi; done | |
| cd <backup_path>/db3/var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE3> "mkdir -p /var/lib/cassandra/data/svc_monitor_keyspace/$dir"; scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE3>:/var/lib/cassandra/data/svc_monitor_keyspace/$dir/; fi; done | |
| 1.3.2.4 Restore initial_tokens | |
| On each of database node, copy the corresponding initial_token from BACKUP_SERVER to /tmp/initial_tokens | |
| DATABASE1: echo "initial_token: $(cat /tmp/initial_tokens)" >> /etc/cassandra/cassandra.yaml | |
| DATABASE2: echo "initial_token: $(cat /tmp/initial_tokens)" >> /etc/cassandra/cassandra.yaml | |
| DATABASE3: echo "initial_token: $(cat /tmp/initial_tokens)" >> /etc/cassandra/cassandra.yaml | |
| FAB_NODE# fab -R database -- "chown -R cassandra:cassandra /var/lib/cassandra/" | |
| 1.3.2.5 Repair and verify database | |
| FAB_NODE# fab start_database | |
| FAB_NODE# fab verify_database | |
| FAB_NODE: fab -R database -- "nodetool repair -- config_db_uuid" | |
| FAB_NODE: fab -R database -- "nodetool repair -- to_bgp_keyspace" | |
| FAB_NODE: fab -R database -- "nodetool repair -- svc_monitor_keyspace" | |
| FAB_NODE# fab restart_cfgm | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment