migrate-gitlab-hosted-to-github.md

Migrate from hosted GitLab to GitHub

In order to migrate from hosted GitLab to GitHub, I decided to use command-line git to mirror each repository from within the GitLab storage on our server.

The script uses the GitHub API to create a new private repository, and then uses the git command-line tool to mirror the repository.

The script keeps track of which repositories have been migrated by creating hidden .migrated-github-<repository> files. If you run the script again, these repositories will be skipped. This means you can re-run the script to recover from errors, such as a repository containing a file larger than 100MB (see https://rtyley.github.io/bfg-repo-cleaner/ for a solution to that).

If you choose to remove those files, you can still run the script multiple times as it mirrors the repository to GitHub. This approach could be used to pick up any changes made in GitLab before the final changeover to GitHub. But see the next paragraph...

WARNING: Mirroring also means that you will replace whatever is in GitHub each time you run this script. So don't run it once you've started to use the GitHub version of a repository.

Preparation

You need to have an SSH key on your server that is connected to your GitHub account.

You need to create a Personal Access Token with access to repo.

Running

Find the directory containing your git repositories. On my system it is in /srv/gitlab/data/git-data/repositories. The script lets you specify a base directory in which to look for bare git repositories, i.e. directories named something.git. So if you use groups in Gitlab, you could choose a group folder to upload.

If you want to upload to an organisation, specify the organisation on the command-line, otherwise omit it.

./migrate-gitlab-hosted-to-github.sh /srv/gitlab/data/git-data/repositories/mygroup <username> <personal access token> [<organisation>]

Notes

Visibility

The script creates private repos. If you want to create public repos, you can edit the script where you see the word "private".

Skipping repos

The script skips the something.wiki.git directories containing the GitLab wikis, as GitHub rejects them.

You can get the script to skip a particular git repository by creating a file in the containing folder .repository_name.ignore, e.g. .myrepository.ignore.

Matching committers

GitHub automatically matches committers based on their email address, against the email addresses validated against the user's GitHub account. You can add additional email addresses to a GitHub account after the repository has been mirrored to GitHub, and GitHub will pick it up and link your commits. It seems to take a few seconds to a minute to catch up.

Repeating the mirroring

If you want to repeat the mirroring for all or some of your repositories, delete the hidden .migrated-github-<repository> files in the directory containing the repositories.

migrate-gitlab-hosted-to-github.sh

#!/bin/bash -eu

basedir=${1:-}
username=${2:-}
password=${3:-}
organisation=${4:-}

if [ -z "$basedir" -o -z "$username" -o -z "$password" ]; then
    echo "usage: $0 <basedir> <username> <password> [<organisation>]" >&2
    exit 1
fi

trap ctrl_c INT

ctrl_c() {
    echo "Interrupted" >&2
    exit 1
}

apiurl=https://api.github.com/user/repos
githubusername="$username"
if [ -n "$organisation" ]; then
    apiurl=https://api.github.com/orgs/$organisation/repos
    githubusername="$organisation"
fi

for dir in $(find "$basedir" -name '*.git' -type d -prune | sort) ; do
    repo="$(basename ${dir%.*})"
    parentdir="$(dirname $dir)"
    
    if [ -f "$parentdir/.migrated-github-$repo" ]; then
        echo "Already migrated: $repo"
        continue
    fi

    # Skip repos that can't be synced
    if [[ "$repo" == *.wiki ]]; then
        echo "Skipping: $repo"
        continue
    fi
    if [ -f "$(dirname $dir)/.$repo.ignore" ]; then
        echo "Ignoring: $repo"
        continue
    fi

    echo "$repo"

    # Create the repo, or ignore if it already exists
    curl -s -o /dev/null -u "$username:$password" $apiurl -d "{\"name\":\"$repo\", \"private\": true}"

    git -C "$dir" push --quiet --no-verify --mirror git@github.com:$githubusername/$repo.git
    touch "$parentdir/.migrated-github-$repo"
done