Skip to content

Instantly share code, notes, and snippets.

@roydejong
Last active February 12, 2024 03:10
Show Gist options
  • Save roydejong/165c72a87da332d1c34b2ec486a7e5bd to your computer and use it in GitHub Desktop.
Save roydejong/165c72a87da332d1c34b2ec486a7e5bd to your computer and use it in GitHub Desktop.
Script: Backup all GitLab repositories to Synology NAS
<?php
// This script will back up all your GitLab repositories to a specified location.
// I recommend creating a seperate GitLab user for backups.
// You'll need to generate a personal access token for that user with API access (in GitLab).
// Next, generate a SSH keypair for the NAS user and attach it to the GitLab user.
// Finally, create a scheduled task in your NAS config to run this script: "php /some/location/git2nas.php"
// Config -- start
$BACKUP_BASEDIR = "/volume1/gitbackup";
$GITLAB_TOKEN = "{SET_ME}";
$GITLAB_SERVER = "{SET_ME}";
// Config -- end
// Step 1: Pull repository list from GitLab API
echo "Connecting to GitLab API to get repository list..." . PHP_EOL;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "{$GITLAB_SERVER}/api/v4/projects?per_page=200");
curl_setopt($ch, CURLOPT_HTTPHEADER, ["PRIVATE-TOKEN: {$GITLAB_TOKEN }"]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
// Step 2: Parse JSON
$parsed = json_decode($response);
$count = count($parsed);
echo "Downloaded list of repositories to back up: found {$count} items" . PHP_EOL;
$i = 0;
// Step 3: Iterate JSON items, cloning or fetching/pulling as needed
foreach ($parsed as $item) {
$sshUrl = $item->ssh_url_to_repo;
$targetPath = $BACKUP_BASEDIR . "/" . $item->path_with_namespace;
$percentage = round(($i / $count) * 100, 2);
$i++;
echo "[{$i} of {$count}, {$percentage}%]" . PHP_EOL;
echo "Cloning {$sshUrl} to {$targetPath}..." . PHP_EOL;
if (!file_exists($targetPath)) {
@shell_exec("mkdir {$targetPath}");
echo shell_exec("cd {$BACKUP_BASEDIR} && git clone {$sshUrl} {$targetPath}") . PHP_EOL;
} else {
echo shell_exec("cd {$targetPath} && git fetch && git pull");
}
}
// That's all, folks!
echo "Completed git backup process";
@Kayu84
Copy link

Kayu84 commented Apr 1, 2021

Thanks for this script, let me comment on your code. Part 1 adjustments, part 2 improvement.


Part1: What would need to be adjusted:
// add requirements line 3:

git is required to clone repositories!
install "git server" from package-center

// only own repositories, I think it is not wanted to pull all public projects
line 20:
?per_page=1000&page=1&membership=true
// recursively folder creation
line 46:
@shell_exec("mkdir -p {$targetPath}");


Part2: I would change the script from a simple clone to a (mirror/bare) repository. This would decrease the disk space requirement, since no working directory is created. Advantage over a pure bare repository, you can still run a fetch.
// clone into .git subfolder only
Line 37:
$targetPath = $BACKUP_BASEDIR . "/" . $item->path_with_namespace . "/.git";
// remove mkdir
line 46: -delete this line-
// clone as --mirror (includes bare)
Line 47:
echo shell_exec("cd {$BACKUP_BASEDIR} && git clone --mirror {$sshUrl} {$targetPath}") . PHP_EOL;
// just fetch, no pull needed (no working directory exists)
Line 49:
echo shell_exec("cd {$targetPath} && git fetch --all");


Additional Info:
I think in a mirrored repository it's also possible to clone from this repository and push the cloned one into the original gitlab source repository... i haven't tested and it could become more complex if on both (source and mirrored) different changes are pushed. could end in special merge conflicts.
But if this mirrored repository is just to hold an additional Backup, this way it saves some disk space.

@Kayu84
Copy link

Kayu84 commented Apr 1, 2021

I've added another Script with just a slight difference:
This script is NOT executed automatically and contains the --prune parameter on fetch.

echo shell_exec("cd {$targetPath} && git fetch --all --prune");

This script is needed if you want to update your NAS-repositories with remotely deleted branches from gitlab.

I've set the script to disabled with time to yesterday and "no repeat". So if unauthorised user will delete branches in gitlab this isn't automatically synced to your NAS-version. So in case i really need to delete branches.
It's not the best solution and i will add a "backup directory" before --prune execution. So always when --prune-script is used i will backup my repositories with "rm -rf backup && cp $BACKUP_BASEDIR backup".
So you can switch back, if the notification email shows there are to many and wrong branches are deleted.
For those two Scripts i've always email-notification on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment