Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Clone all repos from a GitHub organization
curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
@mattheworiordan

This comment has been minimized.

Copy link

commented Sep 12, 2014

Note if you need to access private repos you can modify as follows (replace the [[VARIABLE]] with the suitable value:

curl -u [[USERNAME]] -s https://api.github.com/orgs/[[ORGANIZATION]]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

@mikeyhill

This comment has been minimized.

Copy link

commented Mar 12, 2015

git complained that my rsa key did not exist at gitorious-id_rsa so i simply copied it. Not sure why it would be looking there but it did the trick. Thanks for posting this up.

@gerhardj

This comment has been minimized.

Copy link

commented Mar 17, 2015

i did

sudo apt-get install libjson-ruby

first

@tonyghita

This comment has been minimized.

Copy link

commented Apr 13, 2015

I was able to get access to private repos by creating an OAuth token in the "Personal access tokens" section of the "Applications" tab under "Personal settings", and modifying the snippet to auth with that token:

curl -u <token>:x-oauth-basic -s https://api.github.com/orgs/<organization>/repos\?per_page\=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
@boban-dj

This comment has been minimized.

Copy link

commented Aug 7, 2015

Just to add: it worked for me to backup all my public repo's from user account like this:

curl -u [[USERNAME]] -s https://api.github.com/users/[[USERNAME]]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Thanks for the gist!

@TomHoss

This comment has been minimized.

Copy link

commented Aug 10, 2015

Just tried this and it looks like the max per_page is 100. If you use 200, it will fail silently and cut it off at 100

@muhasturk

This comment has been minimized.

Copy link

commented Oct 4, 2015

I made a script using Python 3 and Github APIv3

https://github.com/muhasturk/gitim

@tiriana

This comment has been minimized.

Copy link

commented Oct 7, 2015

Thank you @caniszczyk, and thank you @mattheworiordan.

@scanf

This comment has been minimized.

Copy link

commented Dec 3, 2015

Thanks, this turned out useful for cloning https://github.com/apple

@subratrout

This comment has been minimized.

Copy link

commented Dec 27, 2015

I tried:
curl -s https://api.github.com/[folder_name]/repos?per_page=20&page=81 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
It gives me following error:
-e:1:in <main>': undefined methodeach' for nil:NilClass (NoMethodError)

@jimhester

This comment has been minimized.

Copy link

commented Feb 26, 2016

If people don't have ruby installed (such as on windows using the git bash prompt) you can use the following.

curl -s https://api.github.com/orgs/twitter/repos\?per_page\=200 | perl -ne 'print "$1\n" if (/"ssh_url": "([^"]+)/)' | xargs -n 1 git clone
@fgimian

This comment has been minimized.

Copy link

commented Jun 1, 2016

Thanks, great job 😄

Here's a version in Python which should work without any extras on both OS X and Linux (as long as Python 2.6 or newer is installed):

curl -s https://api.github.com/users/fgimian/repos?per_page=200 | python -c $'import json, sys, os\nfor repo in json.load(sys.stdin): os.system("git clone " + repo["ssh_url"])'

@erm3nda

This comment has been minimized.

Copy link

commented Jun 4, 2016

Thank you @fgimian, i like to use less dependencies as posible, always. Use the python json module is even better than require to install any other things nor require version 3, as 2.6 is (still) the legacy option.

I found better to use clone_url instead of ssh, wich requires you to have the proper rights. U can use that to clone public repos.
curl -s https://api.github.com/users/fgimian/repos?per_page=200 | python -c $'import json, sys, os\nfor repo in json.load(sys.stdin): os.system("git clone " + repo["clone_url"])'

@1dal

This comment has been minimized.

Copy link

commented Jul 15, 2016

Ok, PHP one-liner here:

php -r 'foreach(json_decode(shell_exec("curl -s https://api.github.com/".readline("Target(\"orgs/twitter\", \"users/onedal88\" etc.): ")."/repos?per_page=31337")) as $r)system("git clone {$r->clone_url}");'

@ameygat

This comment has been minimized.

Copy link

commented Aug 17, 2016

I have written python2 scripts for downloading all repos of a user or a organization Github Python Scripts

@boussou

This comment has been minimized.

Copy link

commented Oct 2, 2016

Hey guys, you don't need python or ruby for that.
here's a pure shell version:

ORGANIZATION=xxx
for i in `curl   -s https://api.github.com/orgs/$ORGANIZATION/repos?per_page=200 |grep html_url|awk 'NR%2 == 0'|cut -d ':'  -f 2-3|tr -d '",'`; do  git clone $i.git;  done

of course, if you put it in a file you should replace $ORGANIZATION by $1

@bnortman

This comment has been minimized.

Copy link

commented Oct 13, 2016

Here is a script I created from these examples that build two additional scripts to manage you Organization Repos

https://github.com/bnortman/useful-utilities/blob/master/bash-scripts/buildSourceTree.sh

@derFunk

This comment has been minimized.

Copy link

commented Dec 2, 2016

This bash script helped me cloning private repos via ssh urls with two factor authentication (using a oauth token - you have to check "Full control of private repositories"):

for i in `curl -u [[USERNAME:TOKEN]] -s "https://api.github.com/orgs/ottonova/repos?per_page=200" |grep ssh_url | cut -d ':' -f 2-3|tr -d '",'`; do git clone $i; done

See here for token generation: https://developer.github.com/v3/auth/#working-with-two-factor-authentication

@brydavis

This comment has been minimized.

Copy link

commented Jan 4, 2017

Thanks for the script!

Ran into permission / SSH key issues on a new computer.

So, I changed script to use "clone_url" instead of "ssh_url".

Here's the full line.

curl -s https://api.github.com/users/brydavis/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["clone_url"]} ]}'

@morgen-peschke

This comment has been minimized.

Copy link

commented Feb 22, 2017

For jq users, here's what it would look like:

curl -u TOKEN:x-oauth-basic 'https://api.github.com/orgs/ORG/repos?per_page=100' | jq '.[].ssh_url' -r | while read url; do git clone "$url"; done
@SISheogorath

This comment has been minimized.

Copy link

commented Mar 9, 2017

And let's simplify it even more:

wget -qO- https://api.github.com/orgs/ORG/repos | jq ".[].ssh_url" | xargs -L 1 git clone
@MgowanoJr

This comment has been minimized.

Copy link

commented Mar 22, 2017

I keep getting this

Permission denied (publickey).
fatal: Could not read from remote repository.

@Phlosioneer

This comment has been minimized.

Copy link

commented Mar 26, 2017

For those having issues with Permission denied (publickey), changing the json key from "ssh_url" to "html_url" will do the trick.

The original line would then become

curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone \"#{repo["html_url"]}\" ]}'

I also found adding --depth 1 right after "git clone" helped speed things up a lot, if you're just trying to quickly search or quickly build the repos.

The original line would then become

curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone --depth 1 #{repo["ssh_url"]} ]}'
@ypid

This comment has been minimized.

Copy link

commented Apr 26, 2017

@Phlosioneer "clone_url" is the proper URL for this case. "html_url has the word "html" in it 😉

@gagomes

This comment has been minimized.

Copy link

commented Jul 7, 2017

My twist on this, with parallel factor of 4 (i.e 4 git clones happening at the same time)

curl -s https://api.github.com/orgs/openstack/repos\?per_page\=200 | grep clone_url | awk -F '"' '{print $4}' | xargs -n 1 -P 4 git clone

@arel

This comment has been minimized.

Copy link

commented Aug 29, 2017

If anyone is looking for this on an enterprise github installation, replace the url with:

  http(s)://hostname/api/v3/orgs/orgname/repos\?per_page\=200 
@bharatdevdas

This comment has been minimized.

Copy link

commented Aug 31, 2017

I'm trying this on our enterprise Git. Thanks to all this solution works.
But, the total number of repositories are more than hundred and even though I pass 'per_page=200' or more, the json returned is always only 100 entries.
Does anyone have a solution?

@thomson

This comment has been minimized.

Copy link

commented Oct 18, 2017

For private repos and paginating through each one steps I took were to create an access token and follow their pagination documentation.

Then, making a request to the organization's repo list endpoint - looking at only the headers you can see the Link header and how to fetch the second page and last page of the organization's repo list.

curl -H 'Authorization: token $TOKEN' -I 'https://api.github.com/orgs/<organization>/repos?per_page=100'

From there, using the link for the next page of results found in the Link header, you can grab the rest of the repos - plugging into the ruby solution shared above:

curl -H "Authorization: token $TOKEN" -s 'https://api.github.com/organizations/<organization id>/repos?per_page=100&page=2' | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
@aterreno

This comment has been minimized.

Copy link

commented Nov 21, 2017

This script btw https://github.com/mazen160/GithubCloner is great and cover most cases

@ciacci1234

This comment has been minimized.

Copy link

commented Dec 11, 2017

@caniszczyk Many thanks for taking the time to post this gist 🌸 I and many others were able to preserve our bootcamp's curriculum for personal use because of it 👍

@nkadel-placester

This comment has been minimized.

Copy link

commented Feb 14, 2018

If you're accessing a private repository for you or for an organization, it can be helpful to replace "curl" with "curl -u username" and type your credentials when running the script. This can be useful to avoid leaving credentials around in your script or a live oauth setup you don't care to leave active.

@atrakic

This comment has been minimized.

Copy link

commented Apr 7, 2018

Awesome. Im using this url to clone starred repos: https://api.github.com/users//starred

@steinfletcher

This comment has been minimized.

Copy link

commented Jun 19, 2018

I made this script, adding support for cloning team repos.

https://github.com/steinfletcher/github-org-clone

@shahzadlone

This comment has been minimized.

Copy link

commented Jun 22, 2018

Here is what i have in my .bashrc (only bash, git, curl needed) that i use as:
$ CloneAll shahzadlone

CloneAll() {
    # Make the url to the input github organization's repository page.
    ORG_URL="https://api.github.com/orgs/${1}/repos?per_page=200";

    # List of all repositories of that organization (seperated by newline-eol).
    ALL_REPOS=$(curl -s ${ORG_URL} | grep html_url | awk 'NR%2 == 0' \
                | cut -d ':' -f 2-3 | tr -d '",');

    # Clone all the repositories.
    for ORG_REPO in ${ALL_REPOS}; do
        git clone ${ORG_REPO}.git;
    done
}
@leighmcculloch

This comment has been minimized.

Copy link

commented Jul 17, 2018

I made this go cli for cloning all my github repos, ignoring archived repos and any already cloned ones:
https://github.com/leighmcculloch/githubcloneall

brew install 4d63/githubcloneall/githubcloneall
githubcloneall -u username
@eallenOP

This comment has been minimized.

Copy link

commented Jul 17, 2018

This took me a while to get right, so I want to add a comment here to help others.

I wanted to clone all the private and public repos in an org that I own. This only worked with 2FA turned off.

curl -u <token>:x-oauth-basic -s https://api.github.com/orgs/<org name>/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone \"#{repo["html_url"]}\" ]}'

I tried lots of different solutions and this was the only way I could get both private and public repos to clone.

@bhilburn

This comment has been minimized.

Copy link

commented Jul 20, 2018

Worked for me on macOS for downloading all repos for an organization

  1. $ brew install jq
  2. Create personal access token on github
  3. $ curl -s https://<your token>:@api.github.com/orgs/<your org>/repos\?per_page\=200 | jq ".[].ssh_url" | xargs -n 1 git clone --recursive
@vSanjo

This comment has been minimized.

Copy link

commented Jul 23, 2018

@leighmcculloch, worked perfectly.

Short, simple and to the point. Thank you. I'll be sure to watch and contribute where I can.

@wuseman

This comment has been minimized.

Copy link

commented Aug 22, 2018

WCLONER

CLONE ALL REPOS FROM A USER WITH LYNX

Hello everyone, thanks alot for all great examples you gave us.

I want to contribute with another very simple example to keep this thread alive.

What is lynx?

Lynx is a text browser for the World Wide Web. Lynx 2.8.8 runs on Un*x, MacOS, VMS, Windows 95/98/NT, DOS386+ (but not 3.1, 3.11), as well as OS/2 EMX. The current developmental version (2.8.9) is also available for testing.

Edit 'U' to the user or organization you wanna mirror all repositories from..

lynx -dump -nonumbers https://github.com/U?tab=repositories|grep '/U/'|cut -d'/' -f1,2,3,4,5|uniq|xargs -L1 git clone

Screenshot

HOW-TO INSTALL LYNX:

Gentoo/Sabayon/Funtoo:

emerge lynx

Debian / Ubuntu / Mint:

apt install lynx -y

Arch Linux

pacman -S lynx 

OpenSUSE/Suse:

zypper in lynx

RHEL Distros:

yum -y install lynx

Android(Termux app required):

pkg install lynx

Windows:

Download exe from here: ftp://ftp.invisible-island.net/lynx/lynx-sl2.8.8rel.1-setup.exe

Macosx:

brew install lynx

FreeBSD/OpenBSD:

 cd /usr/ports/www/lynx/ && make install clean # To install the port
 pkg install lynx # To add the package

Install From Source(Just COPY paste into a shell):

 wget -qP /tmp ftp://ftp.invisible-island.net/lynx/tarballs/lynx2.8.9rel.1.tar.gz; tar -xf /tmp/lynx2.8.9rel.1.tar.gz -C /tmp/; cd /tmp/lynx2.8.9rel.1; ./configure; make; make install

// wuseman

@niranjanreddy891

This comment has been minimized.

Copy link

commented Sep 10, 2018

How to clone all repositories? Can anyone tell me exact solution?

@gabrie30

This comment has been minimized.

Copy link

commented Sep 30, 2018

How to clone all repositories? Can anyone tell me exact solution?

There are some pretty good solutions detailed above. However, you can also try using ghorg which is small cli that will do most of the work for you.

@ericnyamu1

This comment has been minimized.

Copy link

commented Oct 16, 2018

is there away to clone all my starred repos preferably using the access token?

@archenroot

This comment has been minimized.

Copy link

commented Dec 28, 2018

Superior, i like the bashrc solution 👍 or oneliner which can be added into bash_aliases for easy call.

@andreas-bulling

This comment has been minimized.

Copy link

commented Jan 10, 2019

For GitHub Enterprise users:
curl -H 'Authorization: token $TOKEN' -s https://$HOSTNAME/api/v3/orgs/$ORGANISATION/repos\?per_page\=200 | grep clone_url | awk -F '"' '{print $4}' | xargs -n 1 -P 4 git clone

@isPeterXu

This comment has been minimized.

Copy link

commented Feb 10, 2019

curl -u :x-oauth-basic -s https://api.github.com/orgs//repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone "#{repo["html_url"]}" ]}'

How can I get the token?

@gabrie30

This comment has been minimized.

Copy link

commented May 28, 2019

curl -u :x-oauth-basic -s https://api.github.com/orgs//repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone "#{repo["html_url"]}" ]}'

How can I get the token?

If you're on a mac try this $ security find-internet-password -s github.com | grep "acct" | awk -F\" '{ print $4 }'

@bparker06

This comment has been minimized.

Copy link

commented Jun 10, 2019

Another method that doesn't use ruby:

curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | jq -rc '.[] | {clone_url} | .clone_url' | parallel -n 1 git clone {}

It will spawn one clone for every CPU thread you have, and wait until a thread is free before executing another one.

@paravz

This comment has been minimized.

Copy link

commented Jun 28, 2019

with go the added bonus is neat filesystem structure for cloned repos:

curl -sS -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/user/repos?per_page=200 \
	| jq -r '.[].svn_url' \
	| while read svn_url; \
	do go get -d "${svn_url##*://}"; done
@kjenneycloudhealth

This comment has been minimized.

Copy link

commented Jul 11, 2019

To clone ALL repositories for an org (private and public) using an SSH key (which you should be):

 curl -u $USERNAME:$PASSWORD -s https://api.github.com/orgs/$ORG/repos?per_page=200 | jq -rc '.[] | {ssh_url} | .ssh_url' | parallel -n 1 git clone {}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.