Clones all repos in a GitHub organization

clone_all_repos.sh

#!/bin/bash
# Usage: clone_all_repos.sh [organization] <output directory>

ORG=$1
PER_PAGE=100
GIT_OUTPUT_DIRECTORY=${2:-"/tmp/${ORG}_repos"}

if [ -z "$GITHUB_TOKEN" ]; then
    echo -e "Variable GITHUB_TOKEN isn't set! Please specify your GitHub token.\n\nMore info: https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/"
    exit 1
fi

if [ -z "$ORG" ]; then
    echo "Variable ORG isn't set! Please specify the GitHub organization."
    exit 1
fi

mkdir -p $GIT_OUTPUT_DIRECTORY
echo "Cloning repos in $ORG to $GIT_OUTPUT_DIRECTORY/..."

for ((PAGE=1; ; PAGE+=1)); do
    REPO_COUNT=0
    ERROR=0

    while read REPO_NAME ; do
        ((REPO_COUNT++))
        echo -n "Cloning $REPO_NAME to $GIT_OUTPUT_DIRECTORY/$REPONAME... "

        git clone https://github.com/$ORG/$REPO_NAME.git $GIT_OUTPUT_DIRECTORY/$REPO_NAME >/dev/null 2>&1 ||
            { echo -e "ERROR: Unable to clone!" ; ERROR=1 ; continue ; }
        echo "done"
    done < <(curl -u :$GITHUB_TOKEN -s "https://api.github.com/orgs/$ORG/repos?per_page=$PER_PAGE&page=$PAGE" | jq -r ".[]|.name")

    if [ $ERROR -eq 1 ] ; then exit 1 ; fi
    if [ $REPO_COUNT -ne $PER_PAGE ] ; then exit 0 ; fi
done

Doesn't look like you've been rate limited (remaining == limit above).

Did you try simply echoing the repos rather than clone? Maybe try the modified while loop I posted earlier.

echoing the repos made it all the way through..
cloning them only gets to 100 repos...

  sigh
  hzl

ps sorry about the disconnect earlier.. had to rescue a clients network..

turns out when running bash -vx the control flow differed between the count version and the clone version.. and the issue was
{ echo -e "ERROR: Unable to clone!" ; ERROR=1 ; continue ; }

combined with:
if [ $ERROR -eq 1 ] ; then exit 1 ; fi

ended the clone op early.. by changing the line to read :

{ echo -e "ERROR: Unable to clone!" ; ERROR=2 ; continue ; }

and depending on:
if [ $REPO_COUNT -ne $PER_PAGE ] ; then exit 0 ; fi
to break out of the git clone loop the clone all repos succeeds(least its still running...)

hzl

thanx for helping me to safeguard my "bookmarks" ..(my precious)
now since I omitted doing a git remote origin on most of these I am wondering on how to scrape it out of github(have to start reading
that api documentation next :) and add it to all the repos being git cloned..
hzl

This script is awesome. Do you guys mind if I redistribute the gist as part of a repo for bulk cloning git repos (under MIT license)?

have you tried something like this
curl -s https://api.github.com/orgs/blockchain-etl/repos  | jq -r ".[].clone_url" | xargs -L1 git clone

as you already have jq installed

Do you guys mind if I redistribute the gist as part of a repo for bulk cloning git repos (under MIT license)?

Hello @aselunar, sorry for the delay and I appreciate the compliment! Yes, that is fine with me - go ahead and use this as you wish.

@banerRana, yes, that will work for any org that has less than 100 repos. This script handles pagination.