Create a gist now

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Clone all repos from a GitHub organization
curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
@mattheworiordan

This comment has been minimized.

Show comment
Hide comment
@mattheworiordan

mattheworiordan Sep 12, 2014

Note if you need to access private repos you can modify as follows (replace the [[VARIABLE]] with the suitable value:

curl -u [[USERNAME]] -s https://api.github.com/orgs/[[ORGANIZATION]]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Note if you need to access private repos you can modify as follows (replace the [[VARIABLE]] with the suitable value:

curl -u [[USERNAME]] -s https://api.github.com/orgs/[[ORGANIZATION]]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

@mikeyhill

This comment has been minimized.

Show comment
Hide comment
@mikeyhill

mikeyhill Mar 12, 2015

git complained that my rsa key did not exist at gitorious-id_rsa so i simply copied it. Not sure why it would be looking there but it did the trick. Thanks for posting this up.

git complained that my rsa key did not exist at gitorious-id_rsa so i simply copied it. Not sure why it would be looking there but it did the trick. Thanks for posting this up.

@gerhardj

This comment has been minimized.

Show comment
Hide comment
@gerhardj

gerhardj Mar 17, 2015

i did

sudo apt-get install libjson-ruby

first

i did

sudo apt-get install libjson-ruby

first

@tonyghita

This comment has been minimized.

Show comment
Hide comment
@tonyghita

tonyghita Apr 13, 2015

I was able to get access to private repos by creating an OAuth token in the "Personal access tokens" section of the "Applications" tab under "Personal settings", and modifying the snippet to auth with that token:

curl -u <token>:x-oauth-basic -s https://api.github.com/orgs/<organization>/repos\?per_page\=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

I was able to get access to private repos by creating an OAuth token in the "Personal access tokens" section of the "Applications" tab under "Personal settings", and modifying the snippet to auth with that token:

curl -u <token>:x-oauth-basic -s https://api.github.com/orgs/<organization>/repos\?per_page\=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
@boban-dj

This comment has been minimized.

Show comment
Hide comment
@boban-dj

boban-dj Aug 7, 2015

Just to add: it worked for me to backup all my public repo's from user account like this:

curl -u [[USERNAME]] -s https://api.github.com/users/[[USERNAME]]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Thanks for the gist!

boban-dj commented Aug 7, 2015

Just to add: it worked for me to backup all my public repo's from user account like this:

curl -u [[USERNAME]] -s https://api.github.com/users/[[USERNAME]]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Thanks for the gist!

@TomHoss

This comment has been minimized.

Show comment
Hide comment
@TomHoss

TomHoss Aug 10, 2015

Just tried this and it looks like the max per_page is 100. If you use 200, it will fail silently and cut it off at 100

TomHoss commented Aug 10, 2015

Just tried this and it looks like the max per_page is 100. If you use 200, it will fail silently and cut it off at 100

@muhasturk

This comment has been minimized.

Show comment
Hide comment
@muhasturk

muhasturk Oct 4, 2015

I made a script using Python 3 and Github APIv3

https://github.com/muhasturk/gitim

I made a script using Python 3 and Github APIv3

https://github.com/muhasturk/gitim

@tiriana

This comment has been minimized.

Show comment
Hide comment
@tiriana

tiriana Oct 7, 2015

Thank you @caniszczyk, and thank you @mattheworiordan.

tiriana commented Oct 7, 2015

Thank you @caniszczyk, and thank you @mattheworiordan.

@scanf

This comment has been minimized.

Show comment
Hide comment
@scanf

scanf Dec 3, 2015

Thanks, this turned out useful for cloning https://github.com/apple

scanf commented Dec 3, 2015

Thanks, this turned out useful for cloning https://github.com/apple

@subratrout

This comment has been minimized.

Show comment
Hide comment
@subratrout

subratrout Dec 27, 2015

I tried:
curl -s https://api.github.com/[folder_name]/repos?per_page=20&page=81 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
It gives me following error:
-e:1:in <main>': undefined methodeach' for nil:NilClass (NoMethodError)

I tried:
curl -s https://api.github.com/[folder_name]/repos?per_page=20&page=81 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
It gives me following error:
-e:1:in <main>': undefined methodeach' for nil:NilClass (NoMethodError)

@jimhester

This comment has been minimized.

Show comment
Hide comment
@jimhester

jimhester Feb 26, 2016

If people don't have ruby installed (such as on windows using the git bash prompt) you can use the following.

curl -s https://api.github.com/orgs/twitter/repos\?per_page\=200 | perl -ne 'print "$1\n" if (/"ssh_url": "([^"]+)/)' | xargs -n 1 git clone

If people don't have ruby installed (such as on windows using the git bash prompt) you can use the following.

curl -s https://api.github.com/orgs/twitter/repos\?per_page\=200 | perl -ne 'print "$1\n" if (/"ssh_url": "([^"]+)/)' | xargs -n 1 git clone
@fgimian

This comment has been minimized.

Show comment
Hide comment
@fgimian

fgimian Jun 1, 2016

Thanks, great job 😄

Here's a version in Python which should work without any extras on both OS X and Linux (as long as Python 2.6 or newer is installed):

curl -s https://api.github.com/users/fgimian/repos?per_page=200 | python -c $'import json, sys, os\nfor repo in json.load(sys.stdin): os.system("git clone " + repo["ssh_url"])'

fgimian commented Jun 1, 2016

Thanks, great job 😄

Here's a version in Python which should work without any extras on both OS X and Linux (as long as Python 2.6 or newer is installed):

curl -s https://api.github.com/users/fgimian/repos?per_page=200 | python -c $'import json, sys, os\nfor repo in json.load(sys.stdin): os.system("git clone " + repo["ssh_url"])'

@erm3nda

This comment has been minimized.

Show comment
Hide comment
@erm3nda

erm3nda Jun 4, 2016

Thank you @fgimian, i like to use less dependencies as posible, always. Use the python json module is even better than require to install any other things nor require version 3, as 2.6 is (still) the legacy option.

I found better to use clone_url instead of ssh, wich requires you to have the proper rights. U can use that to clone public repos.
curl -s https://api.github.com/users/fgimian/repos?per_page=200 | python -c $'import json, sys, os\nfor repo in json.load(sys.stdin): os.system("git clone " + repo["clone_url"])'

erm3nda commented Jun 4, 2016

Thank you @fgimian, i like to use less dependencies as posible, always. Use the python json module is even better than require to install any other things nor require version 3, as 2.6 is (still) the legacy option.

I found better to use clone_url instead of ssh, wich requires you to have the proper rights. U can use that to clone public repos.
curl -s https://api.github.com/users/fgimian/repos?per_page=200 | python -c $'import json, sys, os\nfor repo in json.load(sys.stdin): os.system("git clone " + repo["clone_url"])'

@1dal

This comment has been minimized.

Show comment
Hide comment
@1dal

1dal Jul 15, 2016

Ok, PHP one-liner here:

php -r 'foreach(json_decode(shell_exec("curl -s https://api.github.com/".readline("Target(\"orgs/twitter\", \"users/onedal88\" etc.): ")."/repos?per_page=31337")) as $r)system("git clone {$r->clone_url}");'

1dal commented Jul 15, 2016

Ok, PHP one-liner here:

php -r 'foreach(json_decode(shell_exec("curl -s https://api.github.com/".readline("Target(\"orgs/twitter\", \"users/onedal88\" etc.): ")."/repos?per_page=31337")) as $r)system("git clone {$r->clone_url}");'

@ameygat

This comment has been minimized.

Show comment
Hide comment
@ameygat

ameygat Aug 17, 2016

I have written python2 scripts for downloading all repos of a user or a organization Github Python Scripts

ameygat commented Aug 17, 2016

I have written python2 scripts for downloading all repos of a user or a organization Github Python Scripts

@boussou

This comment has been minimized.

Show comment
Hide comment
@boussou

boussou Oct 2, 2016

Hey guys, you don't need python or ruby for that.
here's a pure shell version:

ORGANIZATION=xxx
for i in `curl   -s https://api.github.com/orgs/$ORGANIZATION/repos?per_page=200 |grep html_url|awk 'NR%2 == 0'|cut -d ':'  -f 2-3|tr -d '",'`; do  git clone $i.git;  done

of course, if you put it in a file you should replace $ORGANIZATION by $1

boussou commented Oct 2, 2016

Hey guys, you don't need python or ruby for that.
here's a pure shell version:

ORGANIZATION=xxx
for i in `curl   -s https://api.github.com/orgs/$ORGANIZATION/repos?per_page=200 |grep html_url|awk 'NR%2 == 0'|cut -d ':'  -f 2-3|tr -d '",'`; do  git clone $i.git;  done

of course, if you put it in a file you should replace $ORGANIZATION by $1

@bnortman

This comment has been minimized.

Show comment
Hide comment
@bnortman

bnortman Oct 13, 2016

Here is a script I created from these examples that build two additional scripts to manage you Organization Repos

https://github.com/bnortman/useful-utilities/blob/master/bash-scripts/buildSourceTree.sh

Here is a script I created from these examples that build two additional scripts to manage you Organization Repos

https://github.com/bnortman/useful-utilities/blob/master/bash-scripts/buildSourceTree.sh

@derFunk

This comment has been minimized.

Show comment
Hide comment
@derFunk

derFunk Dec 2, 2016

This bash script helped me cloning private repos via ssh urls with two factor authentication (using a oauth token - you have to check "Full control of private repositories"):

for i in `curl -u [[USERNAME:TOKEN]] -s "https://api.github.com/orgs/ottonova/repos?per_page=200" |grep ssh_url | cut -d ':' -f 2-3|tr -d '",'`; do git clone $i; done

See here for token generation: https://developer.github.com/v3/auth/#working-with-two-factor-authentication

derFunk commented Dec 2, 2016

This bash script helped me cloning private repos via ssh urls with two factor authentication (using a oauth token - you have to check "Full control of private repositories"):

for i in `curl -u [[USERNAME:TOKEN]] -s "https://api.github.com/orgs/ottonova/repos?per_page=200" |grep ssh_url | cut -d ':' -f 2-3|tr -d '",'`; do git clone $i; done

See here for token generation: https://developer.github.com/v3/auth/#working-with-two-factor-authentication

@brydavis

This comment has been minimized.

Show comment
Hide comment
@brydavis

brydavis Jan 4, 2017

Thanks for the script!

Ran into permission / SSH key issues on a new computer.

So, I changed script to use "clone_url" instead of "ssh_url".

Here's the full line.

curl -s https://api.github.com/users/brydavis/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["clone_url"]} ]}'

brydavis commented Jan 4, 2017

Thanks for the script!

Ran into permission / SSH key issues on a new computer.

So, I changed script to use "clone_url" instead of "ssh_url".

Here's the full line.

curl -s https://api.github.com/users/brydavis/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["clone_url"]} ]}'

@morgen-peschke

This comment has been minimized.

Show comment
Hide comment
@morgen-peschke

morgen-peschke Feb 22, 2017

For jq users, here's what it would look like:

curl -u TOKEN:x-oauth-basic 'https://api.github.com/orgs/ORG/repos?per_page=100' | jq '.[].ssh_url' -r | while read url; do git clone "$url"; done

For jq users, here's what it would look like:

curl -u TOKEN:x-oauth-basic 'https://api.github.com/orgs/ORG/repos?per_page=100' | jq '.[].ssh_url' -r | while read url; do git clone "$url"; done
@SISheogorath

This comment has been minimized.

Show comment
Hide comment
@SISheogorath

SISheogorath Mar 9, 2017

And let's simplify it even more:

wget -qO- https://api.github.com/orgs/ORG/repos | jq ".[].ssh_url" | xargs -L 1 git clone

SISheogorath commented Mar 9, 2017

And let's simplify it even more:

wget -qO- https://api.github.com/orgs/ORG/repos | jq ".[].ssh_url" | xargs -L 1 git clone
@MgowanoJr

This comment has been minimized.

Show comment
Hide comment
@MgowanoJr

MgowanoJr Mar 22, 2017

I keep getting this

Permission denied (publickey).
fatal: Could not read from remote repository.

I keep getting this

Permission denied (publickey).
fatal: Could not read from remote repository.

@Phlosioneer

This comment has been minimized.

Show comment
Hide comment
@Phlosioneer

Phlosioneer Mar 26, 2017

For those having issues with Permission denied (publickey), changing the json key from "ssh_url" to "html_url" will do the trick.

The original line would then become

curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone \"#{repo["html_url"]}\" ]}'

I also found adding --depth 1 right after "git clone" helped speed things up a lot, if you're just trying to quickly search or quickly build the repos.

The original line would then become

curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone --depth 1 #{repo["ssh_url"]} ]}'

Phlosioneer commented Mar 26, 2017

For those having issues with Permission denied (publickey), changing the json key from "ssh_url" to "html_url" will do the trick.

The original line would then become

curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone \"#{repo["html_url"]}\" ]}'

I also found adding --depth 1 right after "git clone" helped speed things up a lot, if you're just trying to quickly search or quickly build the repos.

The original line would then become

curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone --depth 1 #{repo["ssh_url"]} ]}'
@ypid

This comment has been minimized.

Show comment
Hide comment
@ypid

ypid Apr 26, 2017

@Phlosioneer "clone_url" is the proper URL for this case. "html_url has the word "html" in it 😉

ypid commented Apr 26, 2017

@Phlosioneer "clone_url" is the proper URL for this case. "html_url has the word "html" in it 😉

@gagomes

This comment has been minimized.

Show comment
Hide comment
@gagomes

gagomes Jul 7, 2017

My twist on this, with parallel factor of 4 (i.e 4 git clones happening at the same time)

curl -s https://api.github.com/orgs/openstack/repos\?per_page\=200 | grep clone_url | awk -F '"' '{print $4}' | xargs -n 1 -P 4 git clone

gagomes commented Jul 7, 2017

My twist on this, with parallel factor of 4 (i.e 4 git clones happening at the same time)

curl -s https://api.github.com/orgs/openstack/repos\?per_page\=200 | grep clone_url | awk -F '"' '{print $4}' | xargs -n 1 -P 4 git clone

@arel

This comment has been minimized.

Show comment
Hide comment
@arel

arel Aug 29, 2017

If anyone is looking for this on an enterprise github installation, replace the url with:

  http(s)://hostname/api/v3/orgs/orgname/repos\?per_page\=200 

arel commented Aug 29, 2017

If anyone is looking for this on an enterprise github installation, replace the url with:

  http(s)://hostname/api/v3/orgs/orgname/repos\?per_page\=200 
@bharatdevdas

This comment has been minimized.

Show comment
Hide comment
@bharatdevdas

bharatdevdas Aug 31, 2017

I'm trying this on our enterprise Git. Thanks to all this solution works.
But, the total number of repositories are more than hundred and even though I pass 'per_page=200' or more, the json returned is always only 100 entries.
Does anyone have a solution?

I'm trying this on our enterprise Git. Thanks to all this solution works.
But, the total number of repositories are more than hundred and even though I pass 'per_page=200' or more, the json returned is always only 100 entries.
Does anyone have a solution?

@thomson

This comment has been minimized.

Show comment
Hide comment
@thomson

thomson Oct 18, 2017

For private repos and paginating through each one steps I took were to create an access token and follow their pagination documentation.

Then, making a request to the organization's repo list endpoint - looking at only the headers you can see the Link header and how to fetch the second page and last page of the organization's repo list.

curl -H 'Authorization: token $TOKEN' -I 'https://api.github.com/orgs/<organization>/repos?per_page=100'

From there, using the link for the next page of results found in the Link header, you can grab the rest of the repos - plugging into the ruby solution shared above:

curl -H "Authorization: token $TOKEN" -s 'https://api.github.com/organizations/<organization id>/repos?per_page=100&page=2' | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

thomson commented Oct 18, 2017

For private repos and paginating through each one steps I took were to create an access token and follow their pagination documentation.

Then, making a request to the organization's repo list endpoint - looking at only the headers you can see the Link header and how to fetch the second page and last page of the organization's repo list.

curl -H 'Authorization: token $TOKEN' -I 'https://api.github.com/orgs/<organization>/repos?per_page=100'

From there, using the link for the next page of results found in the Link header, you can grab the rest of the repos - plugging into the ruby solution shared above:

curl -H "Authorization: token $TOKEN" -s 'https://api.github.com/organizations/<organization id>/repos?per_page=100&page=2' | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
@aterreno

This comment has been minimized.

Show comment
Hide comment
@aterreno

aterreno Nov 21, 2017

This script btw https://github.com/mazen160/GithubCloner is great and cover most cases

This script btw https://github.com/mazen160/GithubCloner is great and cover most cases

@ciacci1234

This comment has been minimized.

Show comment
Hide comment
@ciacci1234

ciacci1234 Dec 11, 2017

@caniszczyk Many thanks for taking the time to post this gist 🌸 I and many others were able to preserve our bootcamp's curriculum for personal use because of it 👍

ciacci1234 commented Dec 11, 2017

@caniszczyk Many thanks for taking the time to post this gist 🌸 I and many others were able to preserve our bootcamp's curriculum for personal use because of it 👍

@nkadel-placester

This comment has been minimized.

Show comment
Hide comment
@nkadel-placester

nkadel-placester Feb 14, 2018

If you're accessing a private repository for you or for an organization, it can be helpful to replace "curl" with "curl -u username" and type your credentials when running the script. This can be useful to avoid leaving credentials around in your script or a live oauth setup you don't care to leave active.

If you're accessing a private repository for you or for an organization, it can be helpful to replace "curl" with "curl -u username" and type your credentials when running the script. This can be useful to avoid leaving credentials around in your script or a live oauth setup you don't care to leave active.

@atrakic

This comment has been minimized.

Show comment
Hide comment
@atrakic

atrakic Apr 7, 2018

Awesome. Im using this url to clone starred repos: https://api.github.com/users//starred

atrakic commented Apr 7, 2018

Awesome. Im using this url to clone starred repos: https://api.github.com/users//starred

@steinfletcher

This comment has been minimized.

Show comment
Hide comment
@steinfletcher

steinfletcher Jun 19, 2018

I made this script, adding support for cloning team repos.

https://github.com/steinfletcher/github-org-clone

I made this script, adding support for cloning team repos.

https://github.com/steinfletcher/github-org-clone

@shahzadlone

This comment has been minimized.

Show comment
Hide comment
@shahzadlone

shahzadlone Jun 22, 2018

Here is what i have in my .bashrc (only bash, git, curl needed) that i use as:
$ CloneAll shahzadlone

CloneAll() {
    # Make the url to the input github organization's repository page.
    ORG_URL="https://api.github.com/orgs/${1}/repos?per_page=200";

    # List of all repositories of that organization (seperated by newline-eol).
    ALL_REPOS=$(curl -s ${ORG_URL} | grep html_url | awk 'NR%2 == 0' \
                | cut -d ':' -f 2-3 | tr -d '",');

    # Clone all the repositories.
    for ORG_REPO in ${ALL_REPOS}; do
        git clone ${ORG_REPO}.git;
    done
}

Here is what i have in my .bashrc (only bash, git, curl needed) that i use as:
$ CloneAll shahzadlone

CloneAll() {
    # Make the url to the input github organization's repository page.
    ORG_URL="https://api.github.com/orgs/${1}/repos?per_page=200";

    # List of all repositories of that organization (seperated by newline-eol).
    ALL_REPOS=$(curl -s ${ORG_URL} | grep html_url | awk 'NR%2 == 0' \
                | cut -d ':' -f 2-3 | tr -d '",');

    # Clone all the repositories.
    for ORG_REPO in ${ALL_REPOS}; do
        git clone ${ORG_REPO}.git;
    done
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment