-
-
Save caniszczyk/3856584 to your computer and use it in GitHub Desktop.
| curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}' |
Why do you overcomplicate yourselfs using a lot of stuff, languages and tools? A simple command using bash:
for repo in $(curl -s -iH "Authorization: token ${ACCESS_TOKEN}" https://api.github.com/orgs/${ORG_NAME}/repos?per_page=200 | grep ssh_url | rev | cut -d'"' -f2 | rev); do git clone $repo; doneYou only need to export ACCESS_TOKEN and ORG_NAME
I need the ability to filter and do more than just a single page which is a max 200 repo. Using jq is far more future proof than using cut if the results are changed.
curl -s https://api.github.com/orgs/<ORG_NAME>/repos\?per_page\=100 | jq '.[].html_url' | xargs -n 1 git clone
Thanks man
Is there any way to fork instead of clone all repositories in an organization?
Probably something like https://stackoverflow.com/questions/25446157/creating-a-fork-with-the-github-v3-api-rest
Why do you overcomplicate yourselfs using a lot of stuff, languages and tools? A simple command using bash:
for repo in $(curl -s -iH "Authorization: token ${ACCESS_TOKEN}" https://api.github.com/orgs/${ORG_NAME}/repos?per_page=200 | grep ssh_url | rev | cut -d'"' -f2 | rev); do git clone $repo; doneYou only need to export ACCESS_TOKEN and ORG_NAME
Excellent work, much appreciated.
Create a bash alias/func in your ~/.bashrc or ~/.zshrc file
I solved this for my team by creating an alias/bash func in the ~/.bashrc or ~/.zshrc configuration file.
Config
MacOS Terminal
Open a new Terminal and edit your .bashrc configuration file:
sudo nano ~/.bashrc-- or --
MacOS iTerm2 with oh-my-zsh
Open iTerm2 terminal and edit your .zshrc configuration file:
sudo nano ~/.zshrcCreate CloneAll Func
Add the following bash func as an alias:
CloneAll() {
# Make the url to the input github organization's repository page.
ORG_URL="https://api.github.com/orgs/${1}/repos?per_page=200";
# List of all repositories of that organization (seperated by newline-eol).
ALL_REPOS=$(curl -s ${ORG_URL} | grep html_url | awk 'NR%2 == 0' \
| cut -d ':' -f 2-3 | tr -d '",');
# Clone all the repositories.
for ORG_REPO in ${ALL_REPOS}; do
git clone ${ORG_REPO}.git;
done
}.bashrc
Save and close your ~/.bashrc file and then close the terminal, then open a new one.
You need to do this or the new alias won't initialize:
-- or --
.zshrc
Save and close your ~/.zshrc file and run the following command:
exec "$SHELL"Run
CloneAll <your_github_org_name>Example
If your GitHub repo URL is called https://github.com/flavioespinoza the command would be:
CloneAll flavioespinozaTroubleshooting
Set Number of Repos to Clone
The per_page=200 at the end of the first variable ORG_URL sets the number of repos that will be cloned
ORG_URL="https://api.github.com/orgs/${1}/repos?per_page=<number_of_repos>"; <---- make sure this is what you want
How to run this on Windows10?
btw i want to clone/download all my own repos which include 3 or 4 private repos too.
@caniszczyk
Outdated.
Reason: ubygems is DEAD.
https://www.google.ru/search?q=ubygems
https://blog.engineyard.com/goodbye-ubygems
Parallel, properly paginated (100 per page is max) SSH cloning of public and private org repos - just provide $TOKEN and $ORG:
page=1
while links=($(curl -H "Authorization: token ${TOKEN}" -s "https://api.github.com/orgs/${ORG}/repos?per_page=100&page=${page}" | jq -rc '.[] | {ssh_url} | .ssh_url')); [[ "$links" ]]
do
GIT_TERMINAL_PROMPT=0 parallel git clone --depth=1 {} ::: "${links[@]}"
((++page))
done
I just want to say that this thread is a great example of the open source community. An 8-year thread on a platform, with everybody posting little bits of code and share with others for free, on the platform itself.
I just finished up a simple script if you have a similar oddball setup to me (jq,gh,fish shell). Fish is just used because I like the scripting language. (Not that I'm defensive about fish or anything. haha)
#!/usr/bin/env fish
# You need:
# - fish (https://fishshell.com/) (You don't need to use it as your shell, but it's a good shellscripting language)
# - gh - github cli (https://cli.github.com)
# - jq (https://stedolan.github.io/jq/)
set -q ORG[1]; or begin
echo "I need the org name! I use the environment variable $ORG. example: ORG=octoblu ./download.fish" >&2
exit 1
end
set repos (gh repo list $ORG --limit 9999 --json name)
set repos_to_clone (echo $repos | jq -r ".[].name")
echo "cloning repos:\n $repos_to_clone"
for u in $repos_to_clone
echo "cloning $u"
git clone https://github.com/$ORG/$u repos/$u; or echo "failed to clone $u"
sleep 5
end
fwiw you can also fork all the repos in an org to a new org like this:
#!/usr/bin/env fish
# You need:
# - fish (https://fishshell.com/) (You don't need to use it as your shell, but it's a good shellscripting language)
# - gh - github cli (https://cli.github.com)
# - jq (https://stedolan.github.io/jq/)
set -q OLD_ORG[1]; or begin
echo "I need the org name! I use the environment variable $OLD_ORG. example: OLD_ORG=octoblu NEW_ORG=meshnu ./fork.fish" >&2
exit 1
end
set -q NEW_ORG[1]; or begin
echo "I need the org name! I use the environment variable $NEW_ORG. example: OLD_ORG=octoblu NEW_ORG=meshnu ./fork.fish" >&2
exit 1
end
set repos (gh repo list $OLD_ORG --limit 9999 --json name)
set repos_to_clone (echo $repos | jq -r ".[].name")
echo "cloning repos:\n $repos_to_clone"
for u in $repos_to_clone
echo "forking $u"
gh repo fork $OLD_ORG/$u --clone=false --org $NEW_ORG; or echo "failed to clone $u"
sleep 5
end
# echo $repos
@askfriends ghorg will work on Windows and allows you to clone your own repos as well as from an org
This can also be done with a tool called mergestat, as described on this page of the docs.
For example like so:
mergestat "SELECT clone('https://github.com/mergestat/'|| name) AS path FROM github_org_repos('mergestat')" -v --clone-dir my-dirAn advantage of this approach is that pagination and API rate limits are taken care of by the tool, and arbitrary "sets" of repos can be cloned (filter out by certain criteria such as age or language).
Full disclosure, I am the maintainer/creator of mergestat, but wanted to share as this is a fairly frequent use-case/question we receive from people looking to batch query/work with repos belonging to an org.
Well, I used this repo: https://github.com/muhasturk/gitim
python3 -m gitim -o my_org -t mytoken
Thanks for all the hints. This worked for me and assumes that you've installed gh and jq then authenticated using gh auth login :
gh repo list $ORG --limit 9999 --json sshUrl | jq '.[]|.sshUrl' | xargs -n1 git clone
clone all repos
- you need to install gh (github api)
- windows : choco install gh
- mac : brew install gh
username="ChaitanyaChandra" # user name or org name is required
limit=100
gh repo list $username --limit $limit | awk '{print $1}' > out.txt
while read -r line; do echo "git clone https://github.com/$line.git" >> final.sh ; done < out.txt
chmod +x final.sh
./final.sh
- Into $ORG folder
mkdir $ORG; cd $ORG - Loop clone repo
gh repo list $ORG --limit 9999 --json url | jq '.[]|.url' | xargs -n1 git clone
- Into $ORG folder
mkdir $ORG; cd $ORG- Loop clone repo
gh repo list $ORG --limit 9999 --json url | jq '.[]|.url' | xargs -n1 git clone
Github stopped https clones (username and passwords needed for every clone so instead use ssh clone
gh repo list $ORG --limit 9999 --json sshUrl | jq '.[]|.sshUrl' | xargs -n1 git clone
This also works whilst using gh command and auth instead of git
gh repo list $ORG --limit 9999 --json nameWithOwner | jq '.[]|.nameWithOwner' | xargs -n1 gh repo clone
this can be even simpler, assuming < 100 repos:
curl -s https://api.github.com/orgs/<ORG_NAME>/repos\?per_page\=100 | jq '.[].html_url' | xargs -n 1 git clone
Just in case, if you need ssh url and a bit more than 100 repos:
curl -s "https://api.github.com/orgs/<ORG_NAME>/repos?per_page=100&page=<1,2...>" | jq '.[].ssh_url' | xargs -n 1 git clone
Hi all
I wrote a simple script to clone all the repositories from GitHub, either from a user or whole organization. Please check it out, I'm sure you'll find it useful
Here's a fully working script with things from here, SO + some improvements by me. Needs gh cli.
If the repo exists it will check out the main/master branch and pull.
#!/bin/bash
ORG="myorg" # Your organization
CLONE_PATH="/home/user/Documents/..." # Path in your filesystem where you want to clone the repos
PER_PAGE=100 # per_page maxes out at 100
TOKEN="" # Set your token from GH
for ((PAGE=1; ; PAGE+=1)); do
# Page 0 and 1 are the same
# Change authorization method as needed
INPUT=$(curl -H "Authorization: token $TOKEN" -s "https://api.github.com/orgs/$ORG/repos?per_page=$PER_PAGE&page=$PAGE" | jq -r ".[].clone_url")
if [[ -z "$INPUT" ]]; then
echo "All repos processed, stopped at page=$PAGE"
exit
fi
while read REPO_URL ; do
echo $REPO_URL
temp=${REPO_URL##*/}
repo_name=${temp%.*}
gh repo clone "$REPO_URL" "$CLONE_PATH/$repo_name" -- -q 2>/dev/null || (
cd "$CLONE_PATH/$repo_name"
# Handle case where local checkout is on a non-main/master branch
# - ignore checkout errors because some repos may have zero commits,
# so no main or master
git checkout -q main 2>/dev/null || true
git checkout -q master 2>/dev/null || true
git pull -q
)
done < <(echo "$INPUT")
done
And here's a "Perl-as-glue" version, that works on systems with command-line curl and git: https://gist.github.com/ology/b6087be12131bae0441d40d87505e658 :D
- | xargs -n1 git clone
we can essentially remove jq and directly use gh
gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone
though, it's asking for ssh passcode... any idea how to overcome that?
meh.. I have moved to ghorg. implemented it does everything I could do with my own scripts and is 10 times as fast to boot gh
…
On Mon, Jan 15, 2024 at 10:23 AM Vaisakh K M @.> wrote: @.* commented on this gist. ------------------------------ - | xargs -n1 git clone we can essentially remove jq and directly use gh gh repo list --json=sshUrl --limit 1000 -q ".[].sshUrl" | xargs -n1 git clone — Reply to this email directly, view it on GitHub https://gist.github.com/caniszczyk/3856584#gistcomment-4832274 or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNZZRED6QI6M2V6ZDBT4XTYOVXZXBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFU4ZTQNJWGU4DJJ3UOJUWOZ3FOKTGG4TFMF2GK . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .
thanks.. i didn't know that existed...
Windows / PowerShell
gh repo list theUSERorORG --json=nameWithOwner --limit 1000 -q ".[].nameWithOwner" | %{gh repo clone $_}
Credit to @clrung for the initial version. Minor differences:
Fill out
ORGandTOKEN.Feedback welcome!