Here's a play by play of kenorb's answer to 'How to clone all repos at once from GitHub?' with a breakdown for each command for people (like me) new to bash
TL;DR Run the following command, but replace Kylemit your own Github user name
UserName=Kylemit; \
curl -s https://api.github.com/users/$UserName/repos?per_page=1000 |\
jq -r '.[]|.clone_url' |\
xargs -L1 git cloneOr in a single line:
UserName=Kylemit; curl -s https://api.github.com/users/$UserName/repos?per_page=1000 | jq -r '.[]|.clone_url' | xargs -L1 git cloneNote: You will need to download and alias jq command in order to use the above script.
We'll start with assigning a variable named UserName to the string Kylemit.
Note: There cannot be any spaces between the variable, =, and the value being assigned.
UserName=KylemitOnce the variable has been assigned, you can reference it later by prefixing with a $ like this $UserName
Normally, we can run multiple commands by separating with a line return. But if we want to cram everything into a single line, we can use the ; delimiter.
| symbol | name | description |
|---|---|---|
A ; B |
command delimiter | Run A and then B |
A & B |
parallel control operator | Run both A & B simultaneously |
A && B |
AND control operator | Run B if and only if A succeeded |
A || B |
OR control operator | Run B if and only if A failed |
A | B |
pipe (cmd to cmd redirection) | standard output (stdout) of one command into the standard input (stdin) of another one |
We can get a list of all repositories directly from the User Repos API
GET /users/:username/reposList public repositories for the specified user.
We'll construct the url using variable expansion. If the variable is in a string without quotes or with single quotes, it will be automatically expanded like this:
https://api.github.com/users/$UserName/repos?per_page=1000
https://api.github.com/users/KyleMit/repos?per_page=1000
The command line client for fetching web pages is called cURL
Parameters:
--silent/-s
Silent or quiet mode. Don't show progress meter or error messages.
By setting silent mode, it ensures that the only returned values are from the html response itself
The response is a JSON array of repository objects that look something like this:
{
"id": 158606972,
"node_id": "MDEwOlJlcG9zaXRvcnkxNTg2MDY5NzI=",
"name": "AgileRoadmap",
"full_name": "KyleMit/AgileRoadmap",
"owner": {
"login": "KyleMit",
"url": "https://api.github.com/users/KyleMit",
},
"description": "Tool for building roadmaps and schedules",
"fork": false,
"created_at": "2018-11-21T21:15:20Z",
"url": "https://api.github.com/repos/KyleMit/AgileRoadmap",
"clone_url": "https://github.com/KyleMit/AgileRoadmap.git",
"open_issues": 14,
"default_branch": "master"
},Our next step is to parse the API data that came in as a string of text into something usable. jq is a “filter” command: it takes an input, and produces an output and helps us query json data, but doesn't come standard, so you have to download it first.
After downloading, make sure it's available in your shell by adding to your PATH variable (you might have to restart your shell). One way to make jq available as a command from the downloaded file is to alias it like this:
alias jq="C:/jq-win32.exe"The filter we're using (.[]|.clone_url') consists of several parts:
Array/Object Value Iterator: .[]
If you use the
.[index]syntax, but omit the index entirely, jq will return all of the elements of an array.
Pipe: |
The
|operator combines two filters by feeding the output(s) of the one on the left into the input of the one on the right.
Object Identifier-Index: .foo
The simplest useful filter is .foo. When given a JSON object (aka dictionary or hash) as input, it produces the value at the key “foo”, or null if there’s none present.
--raw-output/-r:
With this option, if the filter’s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.
All told, it'll look like this:
xargs - eXecute command with arguments
Information piped in from a previous argument is available over STDIN (the standard input stream), however not all programs accept commands from standard in and require it being explicitly set in the arguments. Enter xargs, which accepts stdin and passes it along to the following command. For example, the following two commands will execute the same
git clone /path/repo.git
echo /path/repo.git | xargs git cloneSo if we just echo out the input using xargs, we'll get the following:
However, this is kind of broken since we are passing in a long multiline string into the echo command, instead of running echo once for each line of the previous input. We can force xargs to break up the input and parse the input line by line by ussing the parameter -L1
Parameters:
--max-lines[=max-lines]/-L max-lines/-L[max-lines]
Use at most max-lines nonblank input lines per command line.
Which will now pipe each line into our final command
To clone any repository address, we can run the following command, which will copy the repository into a new subfolder with the repo's name within in the current folder.
git clone https://github.com/KyleMit/AgileRoadmap.gitSo before you execute the script, cd your way to where you want your repos setup.
You can make things a little easier on the eyes if you break up each of the commands across seperate lines, but still have them all fire in a single execution (this also provides the benefit of being able to copy and paste the entire script in one motion)
Use a backslash \ for line continuation, and use Shift + Enter while in the editor to create a new line without executing the script. Continued lines, either pasted or typed will be indicated with the > symbol
UserName=Kylemit; \
curl -s https://api.github.com/users/$UserName/repos?per_page=1000 |\
jq -r '.[]|.clone_url' |\
xargs -L1 git cloneOnce the folders have been added locally, in most client side git GUIs you can just drag the repo folders into the editor to add local references to each of those folders to link to them in Source Tree, etc.
