Skip to content

Instantly share code, notes, and snippets.

@KyleMit
Last active May 9, 2020
Embed
What would you like to do?
How to clone all repos at once from GitHub?

Clone all repos from GitHub

Here's a play by play of kenorb's answer to 'How to clone all repos at once from GitHub?' with a breakdown for each command for people (like me) new to bash

TL;DR Run the following command, but replace Kylemit your own Github user name

UserName=Kylemit; \
curl -s https://api.github.com/users/$UserName/repos?per_page=1000 |\
jq -r '.[]|.clone_url' |\
xargs -L1 git clone

Or in a single line:

UserName=Kylemit; curl -s https://api.github.com/users/$UserName/repos?per_page=1000 | jq -r '.[]|.clone_url' | xargs -L1 git clone

Note: You will need to download and alias jq command in order to use the above script.

Variable Assignment

We'll start with assigning a variable named UserName to the string Kylemit.
Note: There cannot be any spaces between the variable, =, and the value being assigned.

UserName=Kylemit

Once the variable has been assigned, you can reference it later by prefixing with a $ like this $UserName

Operators / Separators / Commands

Normally, we can run multiple commands by separating with a line return. But if we want to cram everything into a single line, we can use the ; delimiter.

symbol name description
; B command delimiter Run A and then B
& B parallel control operator Run both A & B simultaneously
&& B AND control operator Run B if and only if A succeeded
|| B OR control operator Run B if and only if A failed
| B pipe (cmd to cmd redirection) standard output (stdout) of one command into the standard input (stdin) of another one

Github Repo API

We can get a list of all repositories directly from the User Repos API

GET /users/:username/repos

List public repositories for the specified user.

Parameter & Variable Substitution/Expansion

We'll construct the url using variable expansion. If the variable is in a string without quotes or with single quotes, it will be automatically expanded like this:

https://api.github.com/users/$UserName/repos?per_page=1000
https://api.github.com/users/KyleMit/repos?per_page=1000

Client URL (cURL)

The command line client for fetching web pages is called cURL

Parameters:

--silent / -s
Silent or quiet mode. Don't show progress meter or error messages.

By setting silent mode, it ensures that the only returned values are from the html response itself

Curl Github

The response is a JSON array of repository objects that look something like this:

{
    "id": 158606972,
    "node_id": "MDEwOlJlcG9zaXRvcnkxNTg2MDY5NzI=",
    "name": "AgileRoadmap",
    "full_name": "KyleMit/AgileRoadmap",
    "owner": {
      "login": "KyleMit",
      "url": "https://api.github.com/users/KyleMit",
    },
    "description": "Tool for building roadmaps and schedules",
    "fork": false,
    "created_at": "2018-11-21T21:15:20Z",
    "url": "https://api.github.com/repos/KyleMit/AgileRoadmap",
    "clone_url": "https://github.com/KyleMit/AgileRoadmap.git",
    "open_issues": 14,
    "default_branch": "master"
  },

jq - JSON Filtering

Demo in jq Fiddle

Our next step is to parse the API data that came in as a string of text into something usable. jq is a “filter” command: it takes an input, and produces an output and helps us query json data, but doesn't come standard, so you have to download it first.

After downloading, make sure it's available in your shell by adding to your PATH variable (you might have to restart your shell). One way to make jq available as a command from the downloaded file is to alias it like this:

alias jq="C:/jq-win32.exe"

The filter we're using (.[]|.clone_url') consists of several parts:

Array/Object Value Iterator: .[]

If you use the .[index] syntax, but omit the index entirely, jq will return all of the elements of an array.

Pipe: |

The | operator combines two filters by feeding the output(s) of the one on the left into the input of the one on the right.

Object Identifier-Index: .foo

The simplest useful filter is .foo. When given a JSON object (aka dictionary or hash) as input, it produces the value at the key “foo”, or null if there’s none present.

Parameters:

--raw-output / -r:
With this option, if the filter’s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.

All told, it'll look like this:

Array Values

xargs - eXecute command with arguments

Information piped in from a previous argument is available over STDIN (the standard input stream), however not all programs accept commands from standard in and require it being explicitly set in the arguments. Enter xargs, which accepts stdin and passes it along to the following command. For example, the following two commands will execute the same

git clone /path/repo.git
echo /path/repo.git | xargs git clone

So if we just echo out the input using xargs, we'll get the following:

xargs whole block

However, this is kind of broken since we are passing in a long multiline string into the echo command, instead of running echo once for each line of the previous input. We can force xargs to break up the input and parse the input line by line by ussing the parameter -L1

Parameters:

--max-lines[=max-lines] / -L max-lines / -L[max-lines]
Use at most max-lines nonblank input lines per command line.

Which will now pipe each line into our final command

xargs line by line

git clone

To clone any repository address, we can run the following command, which will copy the repository into a new subfolder with the repo's name within in the current folder.

git clone https://github.com/KyleMit/AgileRoadmap.git

So before you execute the script, cd your way to where you want your repos setup.

Multi-Line Commands

You can make things a little easier on the eyes if you break up each of the commands across seperate lines, but still have them all fire in a single execution (this also provides the benefit of being able to copy and paste the entire script in one motion)

Use a backslash \ for line continuation, and use Shift + Enter while in the editor to create a new line without executing the script. Continued lines, either pasted or typed will be indicated with the > symbol

UserName=Kylemit; \
curl -s https://api.github.com/users/$UserName/repos?per_page=1000 |\
jq -r '.[]|.clone_url' |\
xargs -L1 git clone

Batch Add

Once the folders have been added locally, in most client side git GUIs you can just drag the repo folders into the editor to add local references to each of those folders to link to them in Source Tree, etc.

add reference in Source Tree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment