Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Clone all repos from a GitHub organization
curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
@mattheworiordan

This comment has been minimized.

Copy link

@mattheworiordan mattheworiordan commented Sep 12, 2014

Note if you need to access private repos you can modify as follows (replace the [[VARIABLE]] with the suitable value:

curl -u [[USERNAME]] -s https://api.github.com/orgs/[[ORGANIZATION]]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

@mikeyhill

This comment has been minimized.

Copy link

@mikeyhill mikeyhill commented Mar 12, 2015

git complained that my rsa key did not exist at gitorious-id_rsa so i simply copied it. Not sure why it would be looking there but it did the trick. Thanks for posting this up.

@gerhardj

This comment has been minimized.

Copy link

@gerhardj gerhardj commented Mar 17, 2015

i did

sudo apt-get install libjson-ruby

first

@tonyghita

This comment has been minimized.

Copy link

@tonyghita tonyghita commented Apr 13, 2015

I was able to get access to private repos by creating an OAuth token in the "Personal access tokens" section of the "Applications" tab under "Personal settings", and modifying the snippet to auth with that token:

curl -u <token>:x-oauth-basic -s https://api.github.com/orgs/<organization>/repos\?per_page\=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
@boban-dj

This comment has been minimized.

Copy link

@boban-dj boban-dj commented Aug 7, 2015

Just to add: it worked for me to backup all my public repo's from user account like this:

curl -u [[USERNAME]] -s https://api.github.com/users/[[USERNAME]]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Thanks for the gist!

@TomHoss

This comment has been minimized.

Copy link

@TomHoss TomHoss commented Aug 10, 2015

Just tried this and it looks like the max per_page is 100. If you use 200, it will fail silently and cut it off at 100

@muhasturk

This comment has been minimized.

Copy link

@muhasturk muhasturk commented Oct 4, 2015

I made a script using Python 3 and Github APIv3

https://github.com/muhasturk/gitim

@tiriana

This comment has been minimized.

Copy link

@tiriana tiriana commented Oct 7, 2015

Thank you @caniszczyk, and thank you @mattheworiordan.

@aalemayhu

This comment has been minimized.

Copy link

@aalemayhu aalemayhu commented Dec 3, 2015

Thanks, this turned out useful for cloning https://github.com/apple

@subratrout

This comment has been minimized.

Copy link

@subratrout subratrout commented Dec 27, 2015

I tried:
curl -s https://api.github.com/[folder_name]/repos?per_page=20&page=81 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
It gives me following error:
-e:1:in <main>': undefined methodeach' for nil:NilClass (NoMethodError)

@jimhester

This comment has been minimized.

Copy link

@jimhester jimhester commented Feb 26, 2016

If people don't have ruby installed (such as on windows using the git bash prompt) you can use the following.

curl -s https://api.github.com/orgs/twitter/repos\?per_page\=200 | perl -ne 'print "$1\n" if (/"ssh_url": "([^"]+)/)' | xargs -n 1 git clone
@fgimian

This comment has been minimized.

Copy link

@fgimian fgimian commented Jun 1, 2016

Thanks, great job 😄

Here's a version in Python which should work without any extras on both OS X and Linux (as long as Python 2.6 or newer is installed):

curl -s https://api.github.com/users/fgimian/repos?per_page=200 | python -c $'import json, sys, os\nfor repo in json.load(sys.stdin): os.system("git clone " + repo["ssh_url"])'

@erm3nda

This comment has been minimized.

Copy link

@erm3nda erm3nda commented Jun 4, 2016

Thank you @fgimian, i like to use less dependencies as posible, always. Use the python json module is even better than require to install any other things nor require version 3, as 2.6 is (still) the legacy option.

I found better to use clone_url instead of ssh, wich requires you to have the proper rights. U can use that to clone public repos.
curl -s https://api.github.com/users/fgimian/repos?per_page=200 | python -c $'import json, sys, os\nfor repo in json.load(sys.stdin): os.system("git clone " + repo["clone_url"])'

@0x6B386F

This comment has been minimized.

Copy link

@0x6B386F 0x6B386F commented Jul 15, 2016

Ok, PHP one-liner here:

php -r 'foreach(json_decode(shell_exec("curl -s https://api.github.com/".readline("Target(\"orgs/twitter\", \"users/onedal88\" etc.): ")."/repos?per_page=31337")) as $r)system("git clone {$r->clone_url}");'

@ameygat

This comment has been minimized.

Copy link

@ameygat ameygat commented Aug 17, 2016

I have written python2 scripts for downloading all repos of a user or a organization Github Python Scripts

@boussou

This comment has been minimized.

Copy link

@boussou boussou commented Oct 2, 2016

Hey guys, you don't need python or ruby for that.
here's a pure shell version:

ORGANIZATION=xxx
for i in `curl   -s https://api.github.com/orgs/$ORGANIZATION/repos?per_page=200 |grep html_url|awk 'NR%2 == 0'|cut -d ':'  -f 2-3|tr -d '",'`; do  git clone $i.git;  done

of course, if you put it in a file you should replace $ORGANIZATION by $1

@bnortman

This comment has been minimized.

Copy link

@bnortman bnortman commented Oct 13, 2016

Here is a script I created from these examples that build two additional scripts to manage you Organization Repos

https://github.com/bnortman/useful-utilities/blob/master/bash-scripts/buildSourceTree.sh

@derFunk

This comment has been minimized.

Copy link

@derFunk derFunk commented Dec 2, 2016

This bash script helped me cloning private repos via ssh urls with two factor authentication (using a oauth token - you have to check "Full control of private repositories"):

for i in `curl -u [[USERNAME:TOKEN]] -s "https://api.github.com/orgs/ottonova/repos?per_page=200" |grep ssh_url | cut -d ':' -f 2-3|tr -d '",'`; do git clone $i; done

See here for token generation: https://developer.github.com/v3/auth/#working-with-two-factor-authentication

@brydavis

This comment has been minimized.

Copy link

@brydavis brydavis commented Jan 4, 2017

Thanks for the script!

Ran into permission / SSH key issues on a new computer.

So, I changed script to use "clone_url" instead of "ssh_url".

Here's the full line.

curl -s https://api.github.com/users/brydavis/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["clone_url"]} ]}'

@morgen-peschke

This comment has been minimized.

Copy link

@morgen-peschke morgen-peschke commented Feb 22, 2017

For jq users, here's what it would look like:

curl -u TOKEN:x-oauth-basic 'https://api.github.com/orgs/ORG/repos?per_page=100' | jq '.[].ssh_url' -r | while read url; do git clone "$url"; done
@SISheogorath

This comment has been minimized.

Copy link

@SISheogorath SISheogorath commented Mar 9, 2017

And let's simplify it even more:

wget -qO- https://api.github.com/orgs/ORG/repos | jq ".[].ssh_url" | xargs -L 1 git clone
@MgowanoJr

This comment has been minimized.

Copy link

@MgowanoJr MgowanoJr commented Mar 22, 2017

I keep getting this

Permission denied (publickey).
fatal: Could not read from remote repository.

@Phlosioneer

This comment has been minimized.

Copy link

@Phlosioneer Phlosioneer commented Mar 26, 2017

For those having issues with Permission denied (publickey), changing the json key from "ssh_url" to "html_url" will do the trick.

The original line would then become

curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone \"#{repo["html_url"]}\" ]}'

I also found adding --depth 1 right after "git clone" helped speed things up a lot, if you're just trying to quickly search or quickly build the repos.

The original line would then become

curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone --depth 1 #{repo["ssh_url"]} ]}'
@ypid

This comment has been minimized.

Copy link

@ypid ypid commented Apr 26, 2017

@Phlosioneer "clone_url" is the proper URL for this case. "html_url has the word "html" in it 😉

@gagomes

This comment has been minimized.

Copy link

@gagomes gagomes commented Jul 7, 2017

My twist on this, with parallel factor of 4 (i.e 4 git clones happening at the same time)

curl -s https://api.github.com/orgs/openstack/repos\?per_page\=200 | grep clone_url | awk -F '"' '{print $4}' | xargs -n 1 -P 4 git clone

@arel

This comment has been minimized.

Copy link

@arel arel commented Aug 29, 2017

If anyone is looking for this on an enterprise github installation, replace the url with:

  http(s)://hostname/api/v3/orgs/orgname/repos\?per_page\=200 
@bharatdevdas

This comment has been minimized.

Copy link

@bharatdevdas bharatdevdas commented Aug 31, 2017

I'm trying this on our enterprise Git. Thanks to all this solution works.
But, the total number of repositories are more than hundred and even though I pass 'per_page=200' or more, the json returned is always only 100 entries.
Does anyone have a solution?

@thomson

This comment has been minimized.

Copy link

@thomson thomson commented Oct 18, 2017

For private repos and paginating through each one steps I took were to create an access token and follow their pagination documentation.

Then, making a request to the organization's repo list endpoint - looking at only the headers you can see the Link header and how to fetch the second page and last page of the organization's repo list.

curl -H 'Authorization: token $TOKEN' -I 'https://api.github.com/orgs/<organization>/repos?per_page=100'

From there, using the link for the next page of results found in the Link header, you can grab the rest of the repos - plugging into the ruby solution shared above:

curl -H "Authorization: token $TOKEN" -s 'https://api.github.com/organizations/<organization id>/repos?per_page=100&page=2' | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
@aterreno

This comment has been minimized.

Copy link

@aterreno aterreno commented Nov 21, 2017

This script btw https://github.com/mazen160/GithubCloner is great and cover most cases

@ciacci1234

This comment has been minimized.

Copy link

@ciacci1234 ciacci1234 commented Dec 11, 2017

@caniszczyk Many thanks for taking the time to post this gist 🌸 I and many others were able to preserve our bootcamp's curriculum for personal use because of it 👍

@nkadel-placester

This comment has been minimized.

Copy link

@nkadel-placester nkadel-placester commented Feb 14, 2018

If you're accessing a private repository for you or for an organization, it can be helpful to replace "curl" with "curl -u username" and type your credentials when running the script. This can be useful to avoid leaving credentials around in your script or a live oauth setup you don't care to leave active.

@atrakic

This comment has been minimized.

Copy link

@atrakic atrakic commented Apr 7, 2018

Awesome. Im using this url to clone starred repos: https://api.github.com/users//starred

@steinfletcher

This comment has been minimized.

Copy link

@steinfletcher steinfletcher commented Jun 19, 2018

I made this script, adding support for cloning team repos.

https://github.com/steinfletcher/github-org-clone

@shahzadlone

This comment has been minimized.

Copy link

@shahzadlone shahzadlone commented Jun 22, 2018

Here is what i have in my .bashrc (only bash, git, curl needed) that i use as:
$ CloneAll shahzadlone

CloneAll() {
    # Make the url to the input github organization's repository page.
    ORG_URL="https://api.github.com/orgs/${1}/repos?per_page=200";

    # List of all repositories of that organization (seperated by newline-eol).
    ALL_REPOS=$(curl -s ${ORG_URL} | grep html_url | awk 'NR%2 == 0' \
                | cut -d ':' -f 2-3 | tr -d '",');

    # Clone all the repositories.
    for ORG_REPO in ${ALL_REPOS}; do
        git clone ${ORG_REPO}.git;
    done
}
@leighmcculloch

This comment has been minimized.

Copy link

@leighmcculloch leighmcculloch commented Jul 17, 2018

I made this go cli for cloning all my github repos, ignoring archived repos and any already cloned ones:
https://github.com/leighmcculloch/githubcloneall

brew install 4d63/githubcloneall/githubcloneall
githubcloneall -u username
@eallenOP

This comment has been minimized.

Copy link

@eallenOP eallenOP commented Jul 17, 2018

This took me a while to get right, so I want to add a comment here to help others.

I wanted to clone all the private and public repos in an org that I own. This only worked with 2FA turned off.

curl -u <token>:x-oauth-basic -s https://api.github.com/orgs/<org name>/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone \"#{repo["html_url"]}\" ]}'

I tried lots of different solutions and this was the only way I could get both private and public repos to clone.

@bhilburn

This comment has been minimized.

Copy link

@bhilburn bhilburn commented Jul 20, 2018

Worked for me on macOS for downloading all repos for an organization

  1. $ brew install jq
  2. Create personal access token on github
  3. $ curl -s https://<your token>:@api.github.com/orgs/<your org>/repos\?per_page\=200 | jq ".[].ssh_url" | xargs -n 1 git clone --recursive
@vSanjo

This comment has been minimized.

Copy link

@vSanjo vSanjo commented Jul 23, 2018

@leighmcculloch, worked perfectly.

Short, simple and to the point. Thank you. I'll be sure to watch and contribute where I can.

@wuseman

This comment has been minimized.

Copy link

@wuseman wuseman commented Aug 22, 2018

WCLONER

CLONE ALL REPOS FROM A USER WITH LYNX

Hello everyone, thanks alot for all great examples you gave us.

I want to contribute with another very simple example to keep this thread alive.

What is lynx?

Lynx is a text browser for the World Wide Web. Lynx 2.8.8 runs on Un*x, MacOS, VMS, Windows 95/98/NT, DOS386+ (but not 3.1, 3.11), as well as OS/2 EMX. The current developmental version (2.8.9) is also available for testing.

Edit 'U' to the user or organization you wanna mirror all repositories from..

lynx -dump -nonumbers https://github.com/U?tab=repositories|grep '/U/'|cut -d'/' -f1,2,3,4,5|uniq|xargs -L1 git clone

Screenshot

HOW-TO INSTALL LYNX:

Gentoo/Sabayon/Funtoo:

emerge lynx

Debian / Ubuntu / Mint:

apt install lynx -y

Arch Linux

pacman -S lynx 

OpenSUSE/Suse:

zypper in lynx

RHEL Distros:

yum -y install lynx

Android(Termux app required):

pkg install lynx

Windows:

Download exe from here: ftp://ftp.invisible-island.net/lynx/lynx-sl2.8.8rel.1-setup.exe

Macosx:

brew install lynx

FreeBSD/OpenBSD:

 cd /usr/ports/www/lynx/ && make install clean # To install the port
 pkg install lynx # To add the package

Install From Source(Just COPY paste into a shell):

 wget -qP /tmp ftp://ftp.invisible-island.net/lynx/tarballs/lynx2.8.9rel.1.tar.gz; tar -xf /tmp/lynx2.8.9rel.1.tar.gz -C /tmp/; cd /tmp/lynx2.8.9rel.1; ./configure; make; make install

// wuseman

@niranjanreddy891

This comment has been minimized.

Copy link

@niranjanreddy891 niranjanreddy891 commented Sep 10, 2018

How to clone all repositories? Can anyone tell me exact solution?

@gabrie30

This comment has been minimized.

Copy link

@gabrie30 gabrie30 commented Sep 30, 2018

How to clone all repositories? Can anyone tell me exact solution?

There are some pretty good solutions detailed above. However, you can also try using ghorg which is small cli that will do most of the work for you.

@archenroot

This comment has been minimized.

Copy link

@archenroot archenroot commented Dec 28, 2018

Superior, i like the bashrc solution 👍 or oneliner which can be added into bash_aliases for easy call.

@andreas-bulling

This comment has been minimized.

Copy link

@andreas-bulling andreas-bulling commented Jan 10, 2019

For GitHub Enterprise users:
curl -H 'Authorization: token $TOKEN' -s https://$HOSTNAME/api/v3/orgs/$ORGANISATION/repos\?per_page\=200 | grep clone_url | awk -F '"' '{print $4}' | xargs -n 1 -P 4 git clone

@isPeterXu

This comment has been minimized.

Copy link

@isPeterXu isPeterXu commented Feb 10, 2019

curl -u :x-oauth-basic -s https://api.github.com/orgs//repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone "#{repo["html_url"]}" ]}'

How can I get the token?

@gabrie30

This comment has been minimized.

Copy link

@gabrie30 gabrie30 commented May 28, 2019

curl -u :x-oauth-basic -s https://api.github.com/orgs//repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone "#{repo["html_url"]}" ]}'

How can I get the token?

If you're on a mac try this $ security find-internet-password -s github.com | grep "acct" | awk -F\" '{ print $4 }'

@bparker06

This comment has been minimized.

Copy link

@bparker06 bparker06 commented Jun 10, 2019

Another method that doesn't use ruby:

curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | jq -rc '.[] | {clone_url} | .clone_url' | parallel -n 1 git clone {}

It will spawn one clone for every CPU thread you have, and wait until a thread is free before executing another one.

@paravz

This comment has been minimized.

Copy link

@paravz paravz commented Jun 28, 2019

with go the added bonus is neat filesystem structure for cloned repos:

curl -sS -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/user/repos?per_page=200 \
	| jq -r '.[].svn_url' \
	| while read svn_url; \
	do go get -d "${svn_url##*://}"; done
@kjenneycloudhealth

This comment has been minimized.

Copy link

@kjenneycloudhealth kjenneycloudhealth commented Jul 11, 2019

To clone ALL repositories for an org (private and public) using an SSH key (which you should be):

 curl -u $USERNAME:$PASSWORD -s https://api.github.com/orgs/$ORG/repos?per_page=200 | jq -rc '.[] | {ssh_url} | .ssh_url' | parallel -n 1 git clone {}
@Sicks3c

This comment has been minimized.

Copy link

@Sicks3c Sicks3c commented Sep 8, 2019

1 .Clone all for repos in $(curl -s https://api.github.com/users/$user/repos | jq -r '.[].html_url' | sed 's/\"//g'); do git clone $repos && sleep 5;done
2 .Exclude forked for repos in $(curl -s https://api.github.com/users/$user/repos | jq -r '.[]|select(.fork ==false)|.html_url' | sed 's/\"//g'); do git clone $repos && sleep 5;done

@jaFerrazza

This comment has been minimized.

Copy link

@jaFerrazza jaFerrazza commented Oct 31, 2019

Worked for me on macOS for downloading all repos for an organization

  1. $ brew install jq
  2. Create personal access token on github
  3. $ curl -s https://<your token>:@api.github.com/orgs/<your org>/repos\?per_page\=200 | jq ".[].ssh_url" | xargs -n 1 git clone --recursive

THANK YOU!

@flavioespinoza

This comment has been minimized.

Copy link

@flavioespinoza flavioespinoza commented Dec 3, 2019

For orgs you have access to with private repos:

curl -u <YOUR_GITHUB_USERNAME> -s https://api.github.com/orgs/<ORG_NAME>/repos?per_page=200 | ruby -rubygems -e ’require “json”; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo[“html_url”]} ]}'

It uses the html_url, so you don't need an access_token just enter your github password when prompted.

@Justintime50

This comment has been minimized.

Copy link

@Justintime50 Justintime50 commented Jan 15, 2020

After some searching and frustrations of my own not being able to do this easily, I built out a project that allows for public and private repo cloning of both personal and organization repos. Perfect for grabbing everything you'd need:

https://github.com/Justintime50/github-archive

@hotelzululima

This comment has been minimized.

Copy link

@hotelzululima hotelzululima commented Jan 25, 2020

your version was the only one that even worked partially for moi.. but stopped after 101 repos downloaded?
and ideas?

  thanx in advance
   hzl

Thanks for the script!

Ran into permission / SSH key issues on a new computer.

So, I changed script to use "clone_url" instead of "ssh_url".

Here's the full line.

curl -s https://api.github.com/users/brydavis/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["clone_url"]} ]}'

@Justintime50

This comment has been minimized.

Copy link

@Justintime50 Justintime50 commented Jan 25, 2020

@hotelzululima, the github api only allows for 100 per “page” so you’d either need to run a second one on page 2 for another hundred or you’d need to iterate/loop through a list of pages per 100 till you got all your repos. Private repos require ssh or some other fun configuration, that’s why most are using them in their examples. See the link above your post for some additional info on those two points EDIT: as that script will iterate over however many repos you'd like and provides SSH instructions.

@hotelzululima

This comment has been minimized.

Copy link

@hotelzululima hotelzululima commented Jan 26, 2020

ok I will retry.. thanx!!

@clrung

This comment has been minimized.

Copy link

@clrung clrung commented Jan 27, 2020

Many of these suggestions use per_page=200 (the limit is 100) and do not respect pagination.

Here's the main part of a bash script that I just wrote to clone all repos in an organization, even when there are more than 100.

ORG=[ORGANIZATION HERE]
GIT_OUTPUT_DIRECTORY=${2:-"/tmp/${ORG}_repos"}
PER_PAGE=100

for ((PAGE=1; ; PAGE+=1)); do
    while read REPO_NAME ; do
        git clone https://github.com/$ORG/$REPO_NAME.git $GIT_OUTPUT_DIRECTORY/$REPO_NAME >/dev/null 2>&1 ||
            { echo "ERROR: Unable to clone!" ; continue ; }
        echo "done"
    done < <(curl -u :$GITHUB_TOKEN -s "https://api.github.com/orgs/$ORG/repos?per_page=$PER_PAGE&page=$PAGE" | jq -r ".[]|.name")
done

Dependencies

  • jq
    • brew install jq
  • GitHub token that has repo>public repo permissions

I used jq -r ".[]|.name" instead of jq -r ".[]|.clone_url" because I wanted to git clone the repositories to a specific location (git clone [url] [destination]), and I needed its name to specify its destination.

The complete gist is here:
https://gist.github.com/clrung/75459a9fe954313c57f69d6cdfd502ec

Hope this helps someone — cheers! 🍻

@CapCap

This comment has been minimized.

Copy link

@CapCap CapCap commented Feb 11, 2020

this can be even simpler, assuming < 100 repos:
curl -s https://api.github.com/orgs/<ORG_NAME>/repos\?per_page\=100 | jq '.[].html_url' | xargs -n 1 git clone

@ftntming

This comment has been minimized.

Copy link

@ftntming ftntming commented Feb 15, 2020

Here is what i have in my .bashrc (only bash, git, curl needed) that i use as:
$ CloneAll shahzadlone

CloneAll() {
    # Make the url to the input github organization's repository page.
    ORG_URL="https://api.github.com/orgs/${1}/repos?per_page=200";

    # List of all repositories of that organization (seperated by newline-eol).
    ALL_REPOS=$(curl -s ${ORG_URL} | grep html_url | awk 'NR%2 == 0' \
                | cut -d ':' -f 2-3 | tr -d '",');

    # Clone all the repositories.
    for ORG_REPO in ${ALL_REPOS}; do
        git clone ${ORG_REPO}.git;
    done
}

Great. Thanks

@johnnylambada

This comment has been minimized.

Copy link

@johnnylambada johnnylambada commented May 13, 2020

Here's what I ended up with after modifying @ftmtming's function for my needs. You'll need SSH access to the repo and you'll need to create a personal access token as described here.

function cloner {
   curl -H "Authorization: token $1" -s "https://api.github.com/orgs/$2/repos?per_page=1000" \
       | sed -n '/"ssh_url"/s/.*ssh_url": "\([^"]*\).*/\1/p' \
       | sort -u \
       | xargs -n1 git clone; 
}

Use it like this: cloner <your-access-token> your-organization

@steven-lai

This comment has been minimized.

Copy link

@steven-lai steven-lai commented Jun 9, 2020

Credit to @clrung for the initial version. Minor differences:

  • This will stop once it has iterated through all the repos (curl returns nothing)
  • Added some minor logging
  • Added ability to ignore repos
  • Added repo type (e.g. public or private)
  • Added ability to do test run (no actual cloning)
  • Added some variation of git-clone (e.g. shallow) if you need it

Fill out ORG and TOKEN.

#!/bin/bash

# Modified version of https://gist.github.com/caniszczyk/3856584#gistcomment-3157288

# Sample syntax:
#   foobar\.git
#   foobar\.git|helloworld\.git
IGNORED_REPO=""
IGNORED_REPO_COUNT=0

ORG="TODO"

# per_page maxes out at 100
PER_PAGE=100

REPO_TYPE="private"

TOKEN="TODO"

TEST_RUN=false

for ((PAGE=1; ; PAGE+=1)); do
  # Page 0 and 1 are the same
  # Change authorization method as needed
  INPUT=$(curl -H "Authorization: token $TOKEN" -s "https://api.github.com/orgs/$ORG/repos?type=$REPO_TYPE&per_page=$PER_PAGE&page=$PAGE" | jq -r ".[].clone_url")
  if [[ -z "$INPUT" ]]; then
    echo "All repos processed, ignored $IGNORED_REPO_COUNT repo(s) and stopped at page=$PAGE"
    exit
  fi
  while read REPO_URL ; do
    if [[ "$REPO_URL" =~ $IGNORED_REPO ]]; then
      echo "*** IGNORING $REPO_URL"
      IGNORED_REPO_COUNT=$((IGNORED_REPO_COUNT+1))
    else
      if $TEST_RUN; then
        echo "git clone $REPO_URL"
      else
        #git clone "$REPO_URL" >/dev/null 2>&1 ||   # Pipe stdout and stderr to /dev/null
        #git clone --depth 1 "$REPO_URL" ||         # Shallow clone for faster cloning, within the repo use the following to get the full git history: git pull --unshallow
        git clone "$REPO_URL" ||                    # Vanilla
          { echo "ERROR: Unable to clone $REPO_URL!" ; continue ; }
      fi
    fi
# This syntax works as well /shrug
#  done <<< "$INPUT"
  done < <(echo "$INPUT")
done

Feedback welcome!

@cruizba

This comment has been minimized.

Copy link

@cruizba cruizba commented Jul 15, 2020

Why do you overcomplicate yourselfs using a lot of stuff, languages and tools? A simple command using bash:

for repo in $(curl -s -iH "Authorization: token ${ACCESS_TOKEN}" https://api.github.com/orgs/${ORG_NAME}/repos?per_page=200 | grep ssh_url | rev | cut -d'"' -f2 | rev); do git clone $repo; done

You only need to export ACCESS_TOKEN and ORG_NAME

@steven-lai

This comment has been minimized.

Copy link

@steven-lai steven-lai commented Jul 15, 2020

I need the ability to filter and do more than just a single page which is a max 200 repo. Using jq is far more future proof than using cut if the results are changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.