Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Gitlab: Clone / Pull all projects in a group
#!/usr/bin/env bash
# Documentation
# https://docs.gitlab.com/ce/api/projects.html#list-projects
NAMESPACE="YOUR_NAMESPACE"
BASE_PATH="https://gitlab.example.com/"
PROJECT_SEARCH_PARAM=""
PROJECT_SELECTION="select(.namespace.name == \"$NAMESPACE\")"
PROJECT_PROJECTION="{ "path": .path, "git": .ssh_url_to_repo }"
if [ -z "$GITLAB_PRIVATE_TOKEN" ]; then
echo "Please set the environment variable GITLAB_PRIVATE_TOKEN"
echo "See ${BASE_PATH}profile/account"
exit 1
fi
FILENAME="repos.json"
trap "{ rm -f $FILENAME; }" EXIT
curl -s "${BASE_PATH}api/v3/projects?private_token=$GITLAB_PRIVATE_TOKEN&search=$PROJECT_SEARCH_PARAM&per_page=999" \
| jq --raw-output --compact-output ".[] | $PROJECT_SELECTION | $PROJECT_PROJECTION" > "$FILENAME"
while read repo; do
THEPATH=$(echo "$repo" | jq -r ".path")
GIT=$(echo "$repo" | jq -r ".git")
if [ ! -d "$THEPATH" ]; then
echo "Cloning $THEPATH ( $GIT )"
git clone "$GIT" --quiet &
else
echo "Pulling $THEPATH"
(cd "$THEPATH" && git pull --quiet) &
fi
done < "$FILENAME"
wait
@JonasGroeger

This comment has been minimized.

Copy link
Owner Author

JonasGroeger commented Dec 12, 2016

  • Added Pagination (per_page=999)
  • Spawn Clone / Pull in Subshell ((git clone / pull))
  • Removed default value for PROJECT_SEARCH_PARAM
@JonasGroeger

This comment has been minimized.

Copy link
Owner Author

JonasGroeger commented Dec 19, 2016

  • Refactored $PROJECT_SELECTION and $PROJECT_PROJECTION for easier modification
    • For example: To only sync non-archived projects, use PROJECT_SELECTION="select(.namespace.name == \"$NAMESPACE\" and .archived == false)"
  • Added trap to remove repos.json on error
@Kingbot

This comment has been minimized.

Copy link

Kingbot commented Apr 4, 2017

Hi Jonas. This looks like exactly what I need. Do I just run this script from the directory that I want to clone the projects into?

@jota3

This comment has been minimized.

Copy link

jota3 commented Jul 11, 2017

Hi. Thanks, your script has been very useful.
I found you can simplify the URL you curl :
"${BASE_PATH}api/v3/projects?private_token=$GITLAB_PRIVATE_TOKEN&search=$PROJECT_SEARCH_PARAM&per_page=999"
by :
"${BASE_PATH}api/v3/groups/$NAMESPACE/projects?private_token=$GITLAB_PRIVATE_TOKEN&per_page=999"
and you won't need the PROJECT_SELECTION with this.

@KosratDAhmad

This comment has been minimized.

Copy link

KosratDAhmad commented Aug 15, 2017

how to run this code??

@JonasGroeger

This comment has been minimized.

Copy link
Owner Author

JonasGroeger commented Sep 21, 2017

@jota3: This only works if you do not want to clone ALL projects I think. The PROJECT_SEARCH_PARAM is for only cloning/pulling some projects.

@Kingbot: You need to have a directory layout like this:

$ tree
.
└── sync-projects

after you run it:

tree -L 1
.
├── project-1/
├── project-2/
├── project-3/
└── sync-projects

for example.

@KosratDAhmad: See https://stackoverflow.com/questions/29099456/how-to-clone-all-projects-of-a-group-at-once-in-gitlab/39747861#39747861

@brycehemme

This comment has been minimized.

Copy link

brycehemme commented Sep 27, 2017

It appears that the per_page parameter might not have the intended effect since per_page is a maximum of 100 by default. This requires the addition of the page parameter to iterate over the available pages. The portion with the API call should be updated to call the API by page. Here's a working implementation -

# replace these lines with the below
#curl -s "${BASE_PATH}api/v3/projects?private_token=$GITLAB_PRIVATE_TOKEN&search=$PROJECT_SEARCH_PARAM&per_page=999" \
#    | jq --raw-output --compact-output ".[] | $PROJECT_SELECTION | $PROJECT_PROJECTION" > "$FILENAME"

# clean up any old files since we'll now be appending to the file
[ -e $FILENAME  ] && rm $FILENAME

PAGE_COUNTER=1
while true; do
    echo "Reading page $PAGE_COUNTER"

    CURL_OUT=$(curl -s -k "${GITLAB_URL}api/v3/projects?private_token=$GITLAB_PRIVATE_TOKEN&search=$PROJECT_SEARCH_PARAM&per_page=999&page=$PAGE_COUNTER")
    if [ "$CURL_OUT" == "[]" ]; then break; fi


    echo $CURL_OUT | jq --raw-output --compact-output ".[] | $PROJECT_SELECTION | $PROJECT_PROJECTION" >> "$FILENAME"
    let PAGE_COUNTER++
done
@gerisse

This comment has been minimized.

Copy link

gerisse commented Oct 17, 2017

thanks a lot , very useful.
I would like , after synchronizing all proejcts locallly, delete the projects on my gitlab server account .
I tried "git remote remove" instead git colne in the script , but failed. An idea ?
Thanks

@viperdriver2000

This comment has been minimized.

Copy link

viperdriver2000 commented Jul 1, 2018

Hi,
the script works fine, but i have a problem.
in my gitlab i have a projekt like this:
infrastruktur > backupstorage > misc
The adress is https://gitlab.mydomain.net/infrastruktur/backupstorage/misc
other projekts look like this
roles > misc
The adress is https://gitlab.mydomain.net/roles/misc

when i use the script i have my git folder on my local machine and i would like to have it synced in infrastruktur/backupstorage/misc.
But the script only sync to misc.
so i have a problem when multiple projekts have "misc".
you understand what i mean ?

i dont figure out what i can do.
do you have any ideas ?

@Anjing1993

This comment has been minimized.

Copy link

Anjing1993 commented Aug 27, 2018

Hi,
I have a problem while I running the code that prompting 'jq: error (at :0): Cannot index string with string "namespace"'.
what should I do?
Thanks

@arganzheng

This comment has been minimized.

Copy link

arganzheng commented Feb 27, 2019

I simplify it to this script:

#!/usr/bin/env bash

BASE_PATH="http://10.21.6.54:6060/"

# GITLAB_PRIVATE_TOKEN=Rq6EnxsQyepvUyH8Dv2J

if [ -z "$1" ]
  then
    echo "group name is required."
    exit 1;
fi

GROUP_NAME="$1"

if [ -z "$GITLAB_PRIVATE_TOKEN" ]; then
    echo "Please set the environment variable GITLAB_PRIVATE_TOKEN"
    echo "See ${BASE_PATH}profile/account"
    exit 1
fi

FIELD_NAME="ssh_url_to_repo"

echo "Cloning all git projects in group $GROUP_NAME";

REPO_SSH_URLS=`curl -s "${BASE_PATH}api/v3/groups/$GROUP_NAME/projects?private_token=$GITLAB_PRIVATE_TOKEN&per_page=999" \
   | grep -o "\"$FIELD_NAME\":[^ ,]\+" | awk -F'"' '{print $4}' | grep $GROUP_NAME`

for REPO_SSH_URL in $REPO_SSH_URLS; do
    THEPATH=$(echo "$REPO_SSH_URL" | awk -F'/' '{print $NF}' | awk -F'.' '{print $1}')

    if [ ! -d "$THEPATH" ]; then
        echo "Cloning $THEPATH ( $REPO_SSH_URL )"
        git clone "$REPO_SSH_URL" --quiet &
    else
        echo "Pulling $THEPATH"
        (cd "$THEPATH" && git pull --quiet) &
    fi
done
@tfarmer00

This comment has been minimized.

Copy link

tfarmer00 commented Mar 4, 2019

I simplify it to this script:

REPO_SSH_URLS=`curl -s "${BASE_PATH}api/v3/groups/$GROUP_NAME/projects?private_token=$GITLAB_PRIVATE_TOKEN&per_page=999" \

For GitLab 11.8, v3 API is deprecated and no longer works. Change v3 to v4 in the URL to get it working.

@SHREDDED-WHEAT

This comment has been minimized.

Copy link

SHREDDED-WHEAT commented May 8, 2019

I simplify it to this script:

#!/usr/bin/env bash

BASE_PATH="http://10.21.6.54:6060/"

# GITLAB_PRIVATE_TOKEN=Rq6EnxsQyepvUyH8Dv2J

if [ -z "$1" ]
  then
    echo "group name is required."
    exit 1;
fi

GROUP_NAME="$1"

if [ -z "$GITLAB_PRIVATE_TOKEN" ]; then
    echo "Please set the environment variable GITLAB_PRIVATE_TOKEN"
    echo "See ${BASE_PATH}profile/account"
    exit 1
fi

FIELD_NAME="ssh_url_to_repo"

echo "Cloning all git projects in group $GROUP_NAME";

REPO_SSH_URLS=`curl -s "${BASE_PATH}api/v3/groups/$GROUP_NAME/projects?private_token=$GITLAB_PRIVATE_TOKEN&per_page=999" \
   | grep -o "\"$FIELD_NAME\":[^ ,]\+" | awk -F'"' '{print $4}' | grep $GROUP_NAME`

for REPO_SSH_URL in $REPO_SSH_URLS; do
    THEPATH=$(echo "$REPO_SSH_URL" | awk -F'/' '{print $NF}' | awk -F'.' '{print $1}')

    if [ ! -d "$THEPATH" ]; then
        echo "Cloning $THEPATH ( $REPO_SSH_URL )"
        git clone "$REPO_SSH_URL" --quiet &
    else
        echo "Pulling $THEPATH"
        (cd "$THEPATH" && git pull --quiet) &
    fi
done

Thank you, all of you, for helping me with this. This is what I have so far.

#!/usr/bin/env bash

BASE_PATH="http://192.168.1.156/"

GITLAB_PRIVATE_TOKEN=iZardfn3AWpxsRM1zjzP

if [ -z "$GITLAB_PRIVATE_TOKEN" ]; then
echo "Please set the environment variable for GITLAB_PRIVATE_TOKEN"
echo "See ${BASE_PATH}profile/account"
exit 1
fi

FIELD_NAME="ssh_url_to_repo"

echo "Cloning or pulling updates on all GitLab projects."

REPO_SSH_URLS=curl -s "${BASE_PATH}api/v4/projects?private_token=$GITLAB_PRIVATE_TOKEN&per_page=999" \ | grep -o "\"$FIELD_NAME\":[^ ,]\+" | awk -F'"' '{print $4}'

echo ""
echo $REPO_SSH_URLS
echo ""

for REPO_SSH_URL in $REPO_SSH_URLS; do
echo $REPO_SSH_URL

REPO_DIR=$(echo "$REPO_SSH_URL" | awk -F'/' '{print $NF}' | awk -F'.' '{print $1}')

if [ ! -d "$REPO_DIR" ]; then
    echo "Cloning $REPO_DIR ( $REPO_SSH_URL )"
    git clone "$REPO_SSH_URL" --quiet &
else
    echo "Pulling $REPO_DIR"
    (cd "$REPO_DIR" && git pull --quiet) &
fi

done

How would I delete directories in the current working directory that the script is running that don't have projects in GitLab?

@gabrie30

This comment has been minimized.

Copy link

gabrie30 commented Aug 3, 2019

Lots of good solutions detailed above. However, you can also try using ghorg which is small cli that will do most of the work for you. It supports github, gitlab, and bitbucket.

@romancin

This comment has been minimized.

Copy link

romancin commented Aug 6, 2019

Lots of good solutions detailed above. However, you can also try using ghorg which is small cli that will do most of the work for you. It supports github, gitlab, and bitbucket.

This doesn't seem to work with GitLab Community edition, only gitlab.com, right?

@angristan

This comment has been minimized.

Copy link

angristan commented Aug 21, 2019

I opened an issue on ghorg and you can now specify your gitlab instance: gabrie30/ghorg#41

Also: https://github.com/angristan/gitlab-repo-dl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.