Bulk backup and clone git repositories from a list

.md

Bulk backup and clone git repositories from a list

Get the git repos list

Option A: Generate a list of repos already cloned on a machine

To do this we will use the linux command grep.

Please jump to How to use linux command under Windows? for more information.

grep -oh "url = .*" */.git/config | cut -d " " -f3 > git_repos.txt

1. The command grep will search for the regular expresion "url = .*" inside the file config in any .git folder, one level down from the curren position.

In this hidden folder each git local repository stores the repository URL, inside the config file (no extension).

The arguments o will make grep to return only the matched (non-empty) parts of a matching line.
The argument h will suppress the prefixing of file names on output.

2. Then it whill pass each result line to the command cut, that will split the string onthe character space and then return the third piece, the actual git repo URL.

3. Finally it will append this line to the file git_repos.txt in the current folder (it will automatically create the file).

Option B: Download a list of all repos SSH urls from a project in Bitbucket.uhub.biz.

In this case we want to download the SSH url for all the existing repos inside a proyect in Bitbucket.

1. Run this command to download a JSON file with all the repos information, including the SSH clone url.

   $ curl --request GET \
   --url 'http://bitbucket.org/rest/api/1.0/projects/[[PROJECT_NAME]]/repos/?limit=1000' \
   --header 'authorization: Basic [[AUTH_STRING]]' \
   --header 'content-type: application/json'
   
   

   • Replace [[PROJECT_NAME]] with the parent project name of the repos you want to download the info.
   • Replace [[AUTH_STRING]] with your user email and password, encoded as a base 64 string (ex.: user@mail.com:password)
2. To extract the ssh urls filter the JSON file go to http://jsonpath.com/
   • In filed JSONPath Syntax copy and paste this string: $.values.*.links.clone[?(@.name=="ssh")].href
   • In field JSON copy and paste the JSON result downloaded on step 1.
3. Create a new git_repos.txt file and copy the content of field Evaluation Results.
4. Search and replace any []", character from the list. Also remove all leading spaces on the lines.
5. All done. Your ready to start cloning.

Batch clone git repositories in the target machine

Now we need to copy the git_repos.txt file with the repositories list to the target machine, and batch clone them.

To do so, we will use this command:

$ grep . git_repos.txt | while read line ; do git clone "$line"; done

How this works?

1. The command grep will read all the lines in the git_repos.txt file, and pass each line to the next command.
2. The command while will read each line and execute as a command the string git clone followed by the current line (a repository URL).

---

Apendix

How to use linux command under Windows?

• On Windows 10 you can use the "Windows Subsystem Linux".
• Since you must have Git installed, another option is to use the Git Bash application.
• And yet another option is to install GnuWin32 to add support for linux commands to Windows.

good!