Instantly share code, notes, and snippets.

Embed
What would you like to do?
Clone all repos from a GitHub organization
curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
@mattheworiordan

This comment has been minimized.

mattheworiordan commented Sep 12, 2014

Note if you need to access private repos you can modify as follows (replace the [[VARIABLE]] with the suitable value:

curl -u [[USERNAME]] -s https://api.github.com/orgs/[[ORGANIZATION]]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

@mikeyhill

This comment has been minimized.

mikeyhill commented Mar 12, 2015

git complained that my rsa key did not exist at gitorious-id_rsa so i simply copied it. Not sure why it would be looking there but it did the trick. Thanks for posting this up.

@gerhardj

This comment has been minimized.

gerhardj commented Mar 17, 2015

i did

sudo apt-get install libjson-ruby

first

@tonyghita

This comment has been minimized.

tonyghita commented Apr 13, 2015

I was able to get access to private repos by creating an OAuth token in the "Personal access tokens" section of the "Applications" tab under "Personal settings", and modifying the snippet to auth with that token:

curl -u <token>:x-oauth-basic -s https://api.github.com/orgs/<organization>/repos\?per_page\=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
@boban-dj

This comment has been minimized.

boban-dj commented Aug 7, 2015

Just to add: it worked for me to backup all my public repo's from user account like this:

curl -u [[USERNAME]] -s https://api.github.com/users/[[USERNAME]]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Thanks for the gist!

@TomHoss

This comment has been minimized.

TomHoss commented Aug 10, 2015

Just tried this and it looks like the max per_page is 100. If you use 200, it will fail silently and cut it off at 100

@muhasturk

This comment has been minimized.

muhasturk commented Oct 4, 2015

I made a script using Python 3 and Github APIv3

https://github.com/muhasturk/gitim

@tiriana

This comment has been minimized.

tiriana commented Oct 7, 2015

Thank you @caniszczyk, and thank you @mattheworiordan.

@scanf

This comment has been minimized.

scanf commented Dec 3, 2015

Thanks, this turned out useful for cloning https://github.com/apple

@subratrout

This comment has been minimized.

subratrout commented Dec 27, 2015

I tried:
curl -s https://api.github.com/[folder_name]/repos?per_page=20&page=81 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
It gives me following error:
-e:1:in <main>': undefined methodeach' for nil:NilClass (NoMethodError)

@jimhester

This comment has been minimized.

jimhester commented Feb 26, 2016

If people don't have ruby installed (such as on windows using the git bash prompt) you can use the following.

curl -s https://api.github.com/orgs/twitter/repos\?per_page\=200 | perl -ne 'print "$1\n" if (/"ssh_url": "([^"]+)/)' | xargs -n 1 git clone
@fgimian

This comment has been minimized.

fgimian commented Jun 1, 2016

Thanks, great job 😄

Here's a version in Python which should work without any extras on both OS X and Linux (as long as Python 2.6 or newer is installed):

curl -s https://api.github.com/users/fgimian/repos?per_page=200 | python -c $'import json, sys, os\nfor repo in json.load(sys.stdin): os.system("git clone " + repo["ssh_url"])'

@erm3nda

This comment has been minimized.

erm3nda commented Jun 4, 2016

Thank you @fgimian, i like to use less dependencies as posible, always. Use the python json module is even better than require to install any other things nor require version 3, as 2.6 is (still) the legacy option.

I found better to use clone_url instead of ssh, wich requires you to have the proper rights. U can use that to clone public repos.
curl -s https://api.github.com/users/fgimian/repos?per_page=200 | python -c $'import json, sys, os\nfor repo in json.load(sys.stdin): os.system("git clone " + repo["clone_url"])'

@1dal

This comment has been minimized.

1dal commented Jul 15, 2016

Ok, PHP one-liner here:

php -r 'foreach(json_decode(shell_exec("curl -s https://api.github.com/".readline("Target(\"orgs/twitter\", \"users/onedal88\" etc.): ")."/repos?per_page=31337")) as $r)system("git clone {$r->clone_url}");'

@ameygat

This comment has been minimized.

ameygat commented Aug 17, 2016

I have written python2 scripts for downloading all repos of a user or a organization Github Python Scripts

@boussou

This comment has been minimized.

boussou commented Oct 2, 2016

Hey guys, you don't need python or ruby for that.
here's a pure shell version:

ORGANIZATION=xxx
for i in `curl   -s https://api.github.com/orgs/$ORGANIZATION/repos?per_page=200 |grep html_url|awk 'NR%2 == 0'|cut -d ':'  -f 2-3|tr -d '",'`; do  git clone $i.git;  done

of course, if you put it in a file you should replace $ORGANIZATION by $1

@bnortman

This comment has been minimized.

bnortman commented Oct 13, 2016

Here is a script I created from these examples that build two additional scripts to manage you Organization Repos

https://github.com/bnortman/useful-utilities/blob/master/bash-scripts/buildSourceTree.sh

@derFunk

This comment has been minimized.

derFunk commented Dec 2, 2016

This bash script helped me cloning private repos via ssh urls with two factor authentication (using a oauth token - you have to check "Full control of private repositories"):

for i in `curl -u [[USERNAME:TOKEN]] -s "https://api.github.com/orgs/ottonova/repos?per_page=200" |grep ssh_url | cut -d ':' -f 2-3|tr -d '",'`; do git clone $i; done

See here for token generation: https://developer.github.com/v3/auth/#working-with-two-factor-authentication

@brydavis

This comment has been minimized.

brydavis commented Jan 4, 2017

Thanks for the script!

Ran into permission / SSH key issues on a new computer.

So, I changed script to use "clone_url" instead of "ssh_url".

Here's the full line.

curl -s https://api.github.com/users/brydavis/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["clone_url"]} ]}'

@morgen-peschke

This comment has been minimized.

morgen-peschke commented Feb 22, 2017

For jq users, here's what it would look like:

curl -u TOKEN:x-oauth-basic 'https://api.github.com/orgs/ORG/repos?per_page=100' | jq '.[].ssh_url' -r | while read url; do git clone "$url"; done
@SISheogorath

This comment has been minimized.

SISheogorath commented Mar 9, 2017

And let's simplify it even more:

wget -qO- https://api.github.com/orgs/ORG/repos | jq ".[].ssh_url" | xargs -L 1 git clone
@MgowanoJr

This comment has been minimized.

MgowanoJr commented Mar 22, 2017

I keep getting this

Permission denied (publickey).
fatal: Could not read from remote repository.

@Phlosioneer

This comment has been minimized.

Phlosioneer commented Mar 26, 2017

For those having issues with Permission denied (publickey), changing the json key from "ssh_url" to "html_url" will do the trick.

The original line would then become

curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone \"#{repo["html_url"]}\" ]}'

I also found adding --depth 1 right after "git clone" helped speed things up a lot, if you're just trying to quickly search or quickly build the repos.

The original line would then become

curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone --depth 1 #{repo["ssh_url"]} ]}'
@ypid

This comment has been minimized.

ypid commented Apr 26, 2017

@Phlosioneer "clone_url" is the proper URL for this case. "html_url has the word "html" in it 😉

@gagomes

This comment has been minimized.

gagomes commented Jul 7, 2017

My twist on this, with parallel factor of 4 (i.e 4 git clones happening at the same time)

curl -s https://api.github.com/orgs/openstack/repos\?per_page\=200 | grep clone_url | awk -F '"' '{print $4}' | xargs -n 1 -P 4 git clone

@arel

This comment has been minimized.

arel commented Aug 29, 2017

If anyone is looking for this on an enterprise github installation, replace the url with:

  http(s)://hostname/api/v3/orgs/orgname/repos\?per_page\=200 
@bharatdevdas

This comment has been minimized.

bharatdevdas commented Aug 31, 2017

I'm trying this on our enterprise Git. Thanks to all this solution works.
But, the total number of repositories are more than hundred and even though I pass 'per_page=200' or more, the json returned is always only 100 entries.
Does anyone have a solution?

@thomson

This comment has been minimized.

thomson commented Oct 18, 2017

For private repos and paginating through each one steps I took were to create an access token and follow their pagination documentation.

Then, making a request to the organization's repo list endpoint - looking at only the headers you can see the Link header and how to fetch the second page and last page of the organization's repo list.

curl -H 'Authorization: token $TOKEN' -I 'https://api.github.com/orgs/<organization>/repos?per_page=100'

From there, using the link for the next page of results found in the Link header, you can grab the rest of the repos - plugging into the ruby solution shared above:

curl -H "Authorization: token $TOKEN" -s 'https://api.github.com/organizations/<organization id>/repos?per_page=100&page=2' | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
@aterreno

This comment has been minimized.

aterreno commented Nov 21, 2017

This script btw https://github.com/mazen160/GithubCloner is great and cover most cases

@ciacci1234

This comment has been minimized.

ciacci1234 commented Dec 11, 2017

@caniszczyk Many thanks for taking the time to post this gist 🌸 I and many others were able to preserve our bootcamp's curriculum for personal use because of it 👍

@nkadel-placester

This comment has been minimized.

nkadel-placester commented Feb 14, 2018

If you're accessing a private repository for you or for an organization, it can be helpful to replace "curl" with "curl -u username" and type your credentials when running the script. This can be useful to avoid leaving credentials around in your script or a live oauth setup you don't care to leave active.

@atrakic

This comment has been minimized.

atrakic commented Apr 7, 2018

Awesome. Im using this url to clone starred repos: https://api.github.com/users//starred

@steinfletcher

This comment has been minimized.

steinfletcher commented Jun 19, 2018

I made this script, adding support for cloning team repos.

https://github.com/steinfletcher/github-org-clone

@shahzadlone

This comment has been minimized.

shahzadlone commented Jun 22, 2018

Here is what i have in my .bashrc (only bash, git, curl needed) that i use as:
$ CloneAll shahzadlone

CloneAll() {
    # Make the url to the input github organization's repository page.
    ORG_URL="https://api.github.com/orgs/${1}/repos?per_page=200";

    # List of all repositories of that organization (seperated by newline-eol).
    ALL_REPOS=$(curl -s ${ORG_URL} | grep html_url | awk 'NR%2 == 0' \
                | cut -d ':' -f 2-3 | tr -d '",');

    # Clone all the repositories.
    for ORG_REPO in ${ALL_REPOS}; do
        git clone ${ORG_REPO}.git;
    done
}
@leighmcculloch

This comment has been minimized.

leighmcculloch commented Jul 17, 2018

I made this go cli for cloning all my github repos, ignoring archived repos and any already cloned ones:
https://github.com/leighmcculloch/githubcloneall

brew install 4d63/githubcloneall/githubcloneall
githubcloneall -u username
@eallenOP

This comment has been minimized.

eallenOP commented Jul 17, 2018

This took me a while to get right, so I want to add a comment here to help others.

I wanted to clone all the private and public repos in an org that I own. This only worked with 2FA turned off.

curl -u <token>:x-oauth-basic -s https://api.github.com/orgs/<org name>/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone \"#{repo["html_url"]}\" ]}'

I tried lots of different solutions and this was the only way I could get both private and public repos to clone.

@bhilburn

This comment has been minimized.

bhilburn commented Jul 20, 2018

Worked for me on macOS for downloading all repos for an organization

  1. $ brew install jq
  2. Create personal access token on github
  3. $ curl -s https://<your token>:@api.github.com/orgs/<your org>/repos\?per_page\=200 | jq ".[].ssh_url" | xargs -n 1 git clone --recursive
@vSanjo

This comment has been minimized.

vSanjo commented Jul 23, 2018

@leighmcculloch, worked perfectly.

Short, simple and to the point. Thank you. I'll be sure to watch and contribute where I can.

@wuseman

This comment has been minimized.

wuseman commented Aug 22, 2018

WCLONER

CLONE ALL REPOS FROM A USER WITH LYNX

Hello everyone, thanks alot for all great examples you gave us.

I want to contribute with another very simple example to keep this thread alive.

What is lynx?

Lynx is a text browser for the World Wide Web. Lynx 2.8.8 runs on Un*x, MacOS, VMS, Windows 95/98/NT, DOS386+ (but not 3.1, 3.11), as well as OS/2 EMX. The current developmental version (2.8.9) is also available for testing.

Edit 'U' to the user or organization you wanna mirror all repositories from..

lynx -dump -nonumbers https://github.com/U?tab=repositories|grep '/U/'|cut -d'/' -f1,2,3,4,5|uniq|xargs -L1 git clone

Screenshot

HOW-TO INSTALL LYNX:

Gentoo/Sabayon/Funtoo:

emerge lynx

Debian / Ubuntu / Mint:

apt install lynx -y

Arch Linux

pacman -S lynx 

OpenSUSE/Suse:

zypper in lynx

RHEL Distros:

yum -y install lynx

Android(Termux app required):

pkg install lynx

Windows:

Download exe from here: ftp://ftp.invisible-island.net/lynx/lynx-sl2.8.8rel.1-setup.exe

Macosx:

brew install lynx

FreeBSD/OpenBSD:

 cd /usr/ports/www/lynx/ && make install clean # To install the port
 pkg install lynx # To add the package

Install From Source(Just COPY paste into a shell):

 wget -qP /tmp ftp://ftp.invisible-island.net/lynx/tarballs/lynx2.8.9rel.1.tar.gz; tar -xf /tmp/lynx2.8.9rel.1.tar.gz -C /tmp/; cd /tmp/lynx2.8.9rel.1; ./configure; make; make install

// wuseman

@niranjanreddy891

This comment has been minimized.

niranjanreddy891 commented Sep 10, 2018

How to clone all repositories? Can anyone tell me exact solution?

@gabrie30

This comment has been minimized.

gabrie30 commented Sep 30, 2018

How to clone all repositories? Can anyone tell me exact solution?

There are some pretty good solutions detailed above. However, you can also try using ghorg which is small cli that will do most of the work for you.

@ericnyamu1

This comment has been minimized.

ericnyamu1 commented Oct 16, 2018

is there away to clone all my starred repos preferably using the access token?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment