Migrating repositories between AE5 versions

migration_repository.md

Migrating an Anaconda Enterprise 5 Repository

When faced with the prospect of moving between versions of AE5, you might be wondering what to do about your repository. Mirroring large repos can take hours. Plus, you may have custom channels that platform users have created to share packages. Follow the steps below to make the process of migrating your repository easier.

Before you begin:

Ensure that you have SSH access between both the existing and new clusters, so that files can be moved within and between systems.

The following processes use the root user, but root access is not required to complete the tasks.

On the newly installed cluster (version 5.2.3)

We'll start by preparing the backup location to receive the database backups, then we'll back up the newer anaconda_repository database.

# Create backup directory and go to it
mkdir /opt/anaconda/backup
cd /opt/anaconda/backup

# Set variables for docker and pod information
POSTGRES_POD="$(kubectl get pods | grep postgres | awk '{print $1}')"
DOCKER_CONT_ID="$(kubectl describe pod $POSTGRES_POD | grep 'Container ID:' | awk '{print $3}' | cut -d'/' -f3)"

# Take a backup of the current anaconda_repository database
gravity exec docker exec -i $DOCKER_CONT_ID /bin/bash -c "su - postgres -c 'pg_dump -U postgres -F t anaconda_repository > /var/lib/postgresql/data/anaconda_repository_initial_db.tar'"

# Move the database dump to the backup directory
mv /opt/anaconda/storage/pgdata/anaconda_repository_initial_db.tar /opt/anaconda/backup/

On the original cluster (version 5.2.1)

Now let's back up the database on the original cluster, and move all the appropriate files to the new cluster. Remember, these commands require SSH access between the two systems, setup using SSH keys.

# Enter gravity
gravity enter

# Get the pod and docker container ID of postgres
POSTGRES_POD="$(kubectl get pods | grep postgres | awk '{print $1}')"
DOCKER_CONT_ID="$(kubectl describe pod $POSTGRES_POD | grep 'Container ID:' | awk '{print $3}' | cut -d'/' -f3)"

# Enter the postgres container and take backup of anaconda_repository database
docker exec -i $DOCKER_CONT_ID /bin/bash -c "su - postgres -c 'pg_dump -U postgres -F t anaconda_repository > /var/lib/postgresql/data/anaconda_repository_restore_db.tar'"

# Exit gravity
exit

# Create backup directory
mkdir /opt/anaconda/backup
cd /opt/anaconda/backup

# Move the dump file created into the backup directory
mv /opt/anaconda/storage/pgdata/anaconda_repository_restore_db.tar /opt/anaconda/backup/

# Move the backup for anaconda_repository to the new cluster using scp
scp anaconda_repository_restore_db.tar root@NEW-CLUSTER-IP:/opt/anaconda/backup/

# rsync the repository files on disk to the new cluster
rsync -avr /opt/anaconda/storage/object/anaconda-repository/ root@NEW-CLUSTER-IP:/opt/anaconda/storage/object/anaconda-repository/

On the newly installed cluster (version 5.2.3)

After moving the backup files, follow these steps to complete the process and restore the original database onto the new cluster:

# Move the backup from the original to the appropriate place
mv /opt/anaconda/backup/anaconda_repository_restore_db.tar /opt/anaconda/storage/pgdata/

# Ensure that ownership of the file is correct
chown polkitd:input /opt/anaconda/storage/pgdata/anaconda_repository_restore_db.tar

# Get the pod and docker container ID of postgres
POSTGRES_POD="$(kubectl get pods | grep postgres | awk '{print $1}')"
DOCKER_CONT_ID="$(kubectl describe pod $POSTGRES_POD | grep 'Container ID:' | awk '{print $3}' | cut -d'/' -f3)"

# Restore the anaconda_repository database
gravity exec docker exec -i $DOCKER_CONT_ID /bin/bash -c "su - postgres -c 'pg_restore -U postgres --clean -d anaconda_repository /var/lib/postgresql/data/anaconda_repository_restore_db.tar'"

Now that the files and database have been restored, you'll need to update the configuration file to make the channels accessible. The instructions to do so are documented here: https://enterprise-docs.anaconda.com/en/latest/admin/chan-pkg/mirror.html#configuring-anaconda-enterprise

After saving your configuration changes, you'll need to restart the pods for the change to take effect on the platform.

# Restart the pods and ensure they come back to a running start
kubectl get pods | grep 'ap-' | cut -d' ' -f1 | xargs kubectl delete pods

To verify the repository migration was successful, ensure that the channels and packages are available on the new cluster: https://enterprise-docs.anaconda.com/en/latest/admin/chan-pkg/manage.html. Then, create or open a project on the new cluster. https://enterprise-docs.anaconda.com/en/latest/data-science-workflows/projects/index.html.