Merge multiple git-repos into one where each repo becomes a sub-directory

_merge-git.md

What it does

This script is hacket together to make merging multiple repos into one as easy as possible. It clones each sub-project and then uses git-filter-repo to move each repo into a sub directory. Finally it uses https://github.com/robinst/git-merge-repos for the final merge.

In other words, it takes this:

| project-1 (separate repo)
| - dir-1
|   - file-1.txt
| - README.md
| project-2 (separate repo)
| - dir-2
|   - file-2.txt
| project-3 (separate repo)
| - dir-3
|   - file-3.txt

and turn it into this:

| new-repo
| - project-1 
|   - dir-1
|     - file-1.txt
|   - README.md
| - project-2
|   - dir-2
|     - file-2.txt
| - project-3
|   - dir-3
|     - file-3.txt

With all history of each project retained.

How-to

This script requires Groovy 2.5.x. You can obtain that by using SDKman (see https://sdkman.io)

To get Groovy on your classpath use this command:

sdk install groovy 2.5.614

If you do not have Java on your classpath you can use SDKman to install that too:

sdk install java 11.0.11-open

Now download merge-git.groovy below and make it executable (this is for posix compatible systems only)

Create a repos.txt file with content like the example below. I use the git@_ notation. I don't know if my script works with https://

Now execute

./merge-git repos.txt

The script will create a ./tmp directory in the current directory and make a script inside it called merge.sh

You can change the directory by adding -d myDir and the script name by adding -s myScripg.sh like this

./merge-git repos.txt -d myDir -s myScript.sh repos.sh

The generated script is not executed, but waiting for you to run it. Please review it before executing.

Note: That the script does not push anything back to the original repos. Also note that the merged repos can be found in ./tmp/merged-repo (or your custom directory). This repo does not have a remote, so you need to add that yourself with git remote

Enjoy. If you use this script, please write a comment below.

Disclaimer

YOU ARE FREE TO USE AND MODIFY THIS SCRIPT. IT COMES WITH NO WARANTY OR GUARANTEES WHAT SO EVER. I TAKE NO RESPONSIBILITY IF YOU USE THIS.

merge-git.groovy

#!/usr/bin/env groovy
import groovy.cli.picocli.CliBuilder
import groovy.cli.picocli.OptionAccessor

import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.Paths
import java.nio.file.attribute.PosixFilePermission

def cli = new CliBuilder(usage: "./${this.class.getSimpleName()}.groovy [file]")
cli.with {
    header = "'file' is a list of repos to process. Just plain repo names, no git info"
    h(longOpt: 'help', "Print usage and exit")
    d(longOpt: 'dir', args: 1, "Directory to run in (will be created if it does not exists, defaults to 'tmp' in current directory)")
}
//noinspection GroovyAssignabilityCheck
OptionAccessor options = cli.parse(args)
List<String> arguments = options.arguments()
if (options.h || arguments.size() != 1) {
    cli.usage()
    System.exit(0)
}
Path cd = Paths.get('.').resolve(options.d as String ?: 'tmp')
if (Files.notExists(cd)) {
    Files.createDirectories(cd)
}

Path reposFile = Paths.get(arguments[0]).toAbsolutePath()

if (Files.notExists(reposFile)) {
    println("${reposFile.toAbsolutePath()} does not exist")
    System.exit(-1)
}
Map<String, String> projects = reposFile.toFile().readLines().findAll().collectEntries {
    String name = it.split('/')[1] - '.git'
    [name, it]
}

List<String> commands = ['#/bin/bash', '']

commands << "cd $cd"

commands.addAll(projects.collectMany { name, repo ->
    [
            "git clone $repo $name",
            "cd $name",
            "git-filter-repo --to-subdirectory-filter $name/",
            "cd .."
    ]
})

if(!Files.exists(cd.resolve('git-merge-repos'))) {
    commands << "git clone https://github.com/robinst/git-merge-repos"
}

List<Path> paths = projects*.key.collect {
    cd.resolve(it)
}
String exp = paths.collect { "${it.toAbsolutePath()}:." }.join(' ')
commands << "./git-merge-repos/run.sh $exp"
commands << "rm -rf ./git-merge-repos "
Path cmdFile = cd.resolve('cmd.sh')
cmdFile.toFile().text = commands.join('\n')
Set<PosixFilePermission> permissions = Files.getPosixFilePermissions(cmdFile) << PosixFilePermission.OWNER_EXECUTE
Files.setPosixFilePermissions(cmdFile, permissions)
println "Everything is ready to execute in ${cd}/cmd.sh"

repos.txt

git@github.com:example/project-1.git
git@github.com:example/project-2.git
git@github.com:example/project-3.git