Batch-migrating Gitolite repositories to Gogs

.md

NOTE: This will only work if you are an administrator on your Gogs instance, or if an administrator has enabled local repository importing for all users.

First, save the following as migrate.sh somewhere, and make it executable (chmod +x migrate.sh):

HOSTNAME="git.cryto.net"
BASEPATH="/home/git/old-repositories/projects/joepie91"

OWNER_ID="$1"
CSRF=`cat ./cookies.txt | grep _csrf | cut -f 7`

while read REPO; do
	REPONAME=`echo "$REPO" | sed "s/\.git\$//"`
	curl "https://$HOSTNAME/repo/migrate" \
		-b "./cookies.txt" \
		-H 'origin: null' \
		-H 'content-type: application/x-www-form-urlencoded' \
		-H "authority: $HOSTNAME" \
		--data "_csrf=$CSRF" \
		--data-urlencode "clone_addr=$BASEPATH/$REPO" \
		--data-urlencode "uid=$OWNER_ID" \
		--data-urlencode "auth_username=" \
		--data-urlencode "auth_password=" \
		--data-urlencode "repo_name=$REPONAME" \
		--data-urlencode "description=Automatically migrated from Gitolite"
done

Change HOSTNAME to point at your Gogs installation, and BASEPATH to point at the folder where your Gitolite repositories live on the filesystem. It must be the entire base path - the repository names cannot contain slashes!

Now save the Gogs cookies from your browser as cookies.txt, and create a file (eg. repositories.txt) containing all your repository names, each on a new line. It could look something like this:

project1.git
project2.git
project3.git

After that, run the following command:

cat repositories.txt | ./migrate.sh 1

... where you replace 1 with your User ID on your Gogs instance.

Done!

$ ls repositories/ | ./migrate.sh 1
Invalid csrf token.

where my cookies.txt is:

_csrf	6aWsgaAxWqI6f8TFHylcR6IjDXY6MTQ5NjY3NzAwMTYyMjY0MTQwOA%3D%3D	git.myhost.com	/	git.myhost.com	1496763401	0

How did to ensure correct CSRF token?

I had the same problem: The cookie-stuff failed.

You can use my webtester to execute the migration. This also works if you are not the administrator of your gogs instance.

Installation

See: https://dev.marc.waeckerlin.org/redmine/

e.g. ubuntu:

sudo apt-get install -y wget software-properties-common apt-transport-https
sudo apt-add-repository https://dev.marc.waeckerlin.org/repository
wget -O- https://dev.marc.waeckerlin.org/repository/PublicKey \
   | sudo apt-key add -
sudo apt-get update -y
sudo apt-get install webtester

Usage

run webtester

in tab Setup Script enter:

timeout 600
set URL=https://mrw.sh/
set USERNAME=mwaeckerlin
set PASSWORD=****
set FROMBASE=https://github.com/mwaeckerlin/
set FROMUSER=mwaeckerlin
set FROMPASSWORD=****
set REPO=android-sdk, apache, authentication.js, bind, bitcoind, boar, boar-data, cordova, docker-backup, docker-openslides, docker.js, dockindock, dokuwiki, dokuwiki2wordpress, dovecot, fake-smtp, gekko, ggrwinti, gitlab, globaleaks, guacamole, icaclient, icinga2ido, icingaweb2, jenkins, kimai, lam, ldap-auth, letsencrypt, mailforward, mingw, minidlna, mysql-autodoc, nextcloud, nginx, ocr, open-budget, openldap, owncloud, php-fpm, phpldapadmin, piwigo, postfix, postgrey, pull-backup, redmine, reverse-proxy, safechat, schroots, seafile, sharing-gallery, sks, ssh, stikked, sugarcrm, svn, svnadmin, sync-to-remote, ubuntu-base, video-converter, vsftpd, webtester, xwiki

where:

• URL: url of your gogs installation including trailing slash
• USERNAME: your username in your gogs installation
• PASSWORD: your password in your gogs installation
• FROMBASE: source repository url, i.e. github base (without project name) including trailing slash
• FROMUSER: username on source, i.e. github user name
• FROMPASSWORD: password on source, i.e. github user password
• REPO: comma separated list of all repositories to import

To get a list of all your repositories, replace mwaeckerlin by your user name and run:

wget -qO- 'https://api.github.com/users/mwaeckerlin/repos?per_page=100' | \
    sed -n 's/ *"name": *"\(.*\)", */\1/p' | tr '\n' ',' | sed 's/,$//'

In tab Test Script, paste:

load URL/user/logout
expect load URL
load URLuser/login
expect load URLuser/login
setvalue #user_name -> 'USERNAME'
setvalue #password -> 'PASSWORD'
click button.button.green.ui
expect load URL
for REPO
  load URLrepo/migrate
  expect load URLrepo/migrate
  setvalue #clone_addr -> 'FROMBASEREPO'
  setvalue #auth_username -> 'FROMUSER'
  setvalue #auth_password -> 'FROMPASSWORD'
  setvalue #repo_name -> 'REPO'
  click div.divider.ui+div.field.required.inline+div.field.required.inline+div.field.inline+div.field.inline>div.checkbox.ui>label
  click button.button.green.ui
  sleep 100
  expect load URLUSERNAME/REPO

Hit Run and wait.

In case of failure: Just remove the repositories that have already been migrated from REPO and hit Run again.

I used the API to import repositories from a local filesystem into gitea.
You have to create a token in user-->settings-->applications first in order to use the API.

#!/bin/sh

HOSTNAME="yourhost.com"
BASEPATH="/path/to/dir/containing/repos"
USERID=3

while read REPO; do
    REPONAME=`echo "$REPO" | sed "s/\.git\$//"`
    curl -H 'Content-Type: application/json'  \
         -H 'Authorization: token your-token-string-here ' \
         -d '{"clone_addr": "'$BASEPATH'/'$REPO'", "uid": '$USERID', "repo_name": "'$REPONAME'"}' \
         -X POST http://$HOSTNAME:3000/api/v1/repos/migrate
done

Here is gogit's API documentation: https://github.com/gogits/go-gogs-client/wiki/Repositories