Skip to content

Instantly share code, notes, and snippets.

@mipaca
Forked from eddiewebb/downloadArtifacts.sh
Created October 9, 2019 21:43
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mipaca/f5e5c95ae92f3ba8ce0534731ceed81b to your computer and use it in GitHub Desktop.
Save mipaca/f5e5c95ae92f3ba8ce0534731ceed81b to your computer and use it in GitHub Desktop.
Download CircleCI Artifacts with directory structure
#!/usr/bin/env bash
#
# Download all artitacts for given build, mimicing structure locally.
#
# Batch size is slighly misleading, as it;s only how many URLs to attempt before momentary pause
#
# Can be used on CCI builds with parallelism and will only download the files for matcing index.
#
#
USAGE="$0 username/repo 123 [bitbucket] [batch_size]"
PROJECT=${1} #user/project
BUILD=${2} #1234
VCS=${3:-"github"}
BATCH_SIZE=${4:-100}
: ${CIRCLE_API_TOKEN:?"Please set CIRCLE_API_TOKEN as env variable"}
: ${BUILD:?"Please provide build number, usage: $USAGE"}
case $PROJECT in
*\/*)
;;
*)
echo -e "\nERROR: Invalid project format.\nusage: $USAGE"
exit 1
esac
FILENAME="steps-${BUILD}.txt"
URL="https://circleci.com/api/v1.1/project/${VCS}/${PROJECT}/$BUILD/artifacts"
if [ -s $FILENAME ];then
echo "Using existing list, delete ${FILENAME} to refresh"
else
if [ "${CIRCLE_NODE_TOTAL:-1}" -gt "1" ];then
echo "Running parallel nodes, only download files for index ${CIRCLE_NODE_INDEX}"
curl -s -u $CIRCLE_API_TOKEN: $URL | jq -r --arg token "$CIRCLE_API_TOKEN" --arg build "$BUILD" --argjson node ${CIRCLE_NODE_INDEX} '.[] | select(.node_index==$node) | "mkdir -p $(dirname " + $build + "/\(.node_index)/\(.path)) && curl -s \(.url | @sh)?circle-token=" + $token + " -o " + $build + "/\(.node_index)/\(.path)" ' > $FILENAME
else
curl -s -u $CIRCLE_API_TOKEN: $URL | jq -r --arg token "$CIRCLE_API_TOKEN" --arg build "$BUILD" '.[] | "mkdir -p $(dirname " + $build + "/\(.node_index)/\(.path)) && curl -s \(.url | @sh)?circle-token=" + $token + " -o " + $build + "/\(.node_index)/\(.path)" ' > $FILENAME
fi
fi
echo "Will download $(wc -l ${FILENAME}) files"
current=0
backoff=2
while read cmd; do
current=$(expr $current + 1)
if [ $current -gt $BATCH_SIZE ];then
echo "+$BATCH_SIZE.."
current=0
sleep $backoff
fi
eval $cmd &
done < $FILENAME
@HolimaX
Copy link

HolimaX commented Feb 22, 2022

Any chance to get this updated for CircleCI 2.x ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment