Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Download CircleCI Artifacts with directory structure
#!/usr/bin/env bash
#
# Download all artitacts for given build, mimicing structure locally.
#
# Batch size is slighly misleading, as it;s only how many URLs to attempt before momentary pause
#
# Can be used on CCI builds with parallelism and will only download the files for matcing index.
#
#
USAGE="$0 username/repo 123 [bitbucket] [batch_size]"
PROJECT=${1} #user/project
BUILD=${2} #1234
VCS=${3:-"github"}
BATCH_SIZE=${4:-100}
: ${CIRCLE_API_TOKEN:?"Please set CIRCLE_API_TOKEN as env variable"}
: ${BUILD:?"Please provide build number, usage: $USAGE"}
case $PROJECT in
*\/*)
;;
*)
echo -e "\nERROR: Invalid project format.\nusage: $USAGE"
exit 1
esac
URL="https://circleci.com/api/v1.1/project/${VCS}/${PROJECT}/$BUILD/artifacts"
if [ -s steps.txt ];then
echo "Using existing list, delete steps.txt to refresh"
else
if [ "${CIRCLE_NODE_TOTAL:-1}" -gt "1" ];then
echo "Running parallel nodes, only download files for index ${CIRCLE_NODE_INDEX}"
curl -s -u $CIRCLE_API_TOKEN: $URL | jq -r --argjson node ${CIRCLE_NODE_INDEX} '.[] | select(.node_index==$node) | "mkdir -p $(dirname \(.node_index)/\(.path)) && curl -s \(.url | @sh) -o \(.node_index)/\(.path)" ' > steps.txt
else
curl -s -u $CIRCLE_API_TOKEN: $URL | jq -r '.[] | "mkdir -p $(dirname \(.node_index)/\(.path)) && curl -s \(.url | @sh) -o \(.node_index)/\(.path)" ' > steps.txt
fi
fi
echo "Will download $(wc -l steps.txt) files"
current=0
backoff=2
while read cmd; do
current=$(expr $current + 1)
if [ $current -gt $BATCH_SIZE ];then
echo "+$BATCH_SIZE.."
current=0
sleep $backoff
fi
eval $cmd &
done < steps.txt
@mipaca

This comment has been minimized.

Copy link

mipaca commented Oct 9, 2019

Thanks for the script. I forked it because I found that running it required appending the $CIRCLE_API_TOKEN to each URL. I also wanted to download multiple builds so modified the steps.txt file name and local path to include the $BUILD variable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.