Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Script for removing GitLab Job Artifacts.
#!/bin/bash
#
# Written by Chris Arceneaux
# GitHub: https://github.com/carceneaux
# Email: carcenea@gmail.com
# Website: http://arsano.ninja
#
# Note: This code is a stop-gap to erase Job Artifacts for a project. I HIGHLY recommend you leverage
# "artifacts:expire_in" in your .gitlab-ci.yml
#
# https://docs.gitlab.com/ee/ci/yaml/#artifactsexpire_in
#
# Software Requirements: curl, jq
#
# This code has been released under the terms of the Apache-2.0 license
# http://opensource.org/licenses/Apache-2.0
# project_id, find it here: https://gitlab.com/[organization name]/[repository name] at the top underneath repository name
project_id="207"
# token, find it here: https://gitlab.com/profile/personal_access_tokens
token="9hjGYpwmsMfBxT-Ghuu7"
server="gitlab.com"
# Retrieving Jobs list page count
total_pages=$(curl -sD - -o /dev/null -X GET \
"https://$server/api/v4/projects/$project_id/jobs?per_page=100" \
-H "PRIVATE-TOKEN: ${token}" | grep -Fi X-Total-Pages | sed 's/[^0-9]*//g')
# Creating list of Job IDs for the Project specified with Artifacts
job_ids=()
echo ""
echo "Creating list of all Jobs that currently have Artifacts..."
echo "Total Pages: ${total_pages}"
for ((i=2;i<=${total_pages};i++)) #starting with page 2 skipping most recent 100 Jobs
do
echo "Processing Page: ${i}/${total_pages}"
response=$(curl -s -X GET \
"https://$server/api/v4/projects/$project_id/jobs?per_page=100&page=${i}" \
-H "PRIVATE-TOKEN: ${token}")
length=$(echo $response | jq '. | length')
for ((j=0;j<${length};j++))
do
if [[ $(echo $response | jq ".[${j}].artifacts_file | length") > 0 ]]; then
echo "Job found: $(echo $response | jq ".[${j}].id")"
job_ids+=($(echo $response | jq ".[${j}].id"))
fi
done
done
# Loop through each Job erasing the Artifact(s)
echo ""
echo "${#job_ids[@]} Jobs found. Commencing removal of Artifacts..."
for job_id in ${job_ids[@]};
do
response=$(curl -s -X DELETE \
-H "PRIVATE-TOKEN:${token}" \
"https://$server/api/v4/projects/$project_id/jobs/$job_id/artifacts")
echo "Processing Job ID: ${job_id} - Status: $(echo $response | jq '.status')"
done
@tamasgal

This comment has been minimized.

Copy link

@tamasgal tamasgal commented Sep 13, 2019

Thanks, still works fine on self-hosted GitLab EE 12.2 👍

@carceneaux

This comment has been minimized.

Copy link
Owner Author

@carceneaux carceneaux commented Sep 13, 2019

Thanks! 🍻 Glad to hear!

@YoungPyDawan

This comment has been minimized.

@carceneaux

This comment has been minimized.

Copy link
Owner Author

@carceneaux carceneaux commented Oct 21, 2019

Thanks @YoungPyDawan! I've modified the API call so that only the artifacts are deleted. Good to see that API call was added. 😄

@Kage-Yami

This comment has been minimized.

Copy link

@Kage-Yami Kage-Yami commented Apr 5, 2020

FYI... I came across this today in my search for an easy way to delete all artifacts for a project; unfortunately, it won't work in all cases (like mine) due to X-Total-Pages being omitted when the item count is greater than 10,000.

@carceneaux

This comment has been minimized.

Copy link
Owner Author

@carceneaux carceneaux commented Apr 6, 2020

@Kage-Yami - Thanks for the heads up! I'll work on an updated version of the code. The fix is to not worry about X-Total-Page and simply check for the X-Next-Page header and key off of it instead.

As I'm pretty busy right now, it'll take a week or two for me to get to this. If you get the code sorted before then, please share. 😄

Here's the link mentioning the new logic to be used if you're interested:

https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/23931/diffs#34fe105b9f0ef77edad95de0c13084ff7f54c344_260_298

@Kage-Yami

This comment has been minimized.

Copy link

@Kage-Yami Kage-Yami commented Apr 7, 2020

I ended up writing my own version that accepts the number of pages as an argument (with project and token also being arguments); I manually determined the page count by trial-and-error beforehand. So not great, but it got the job done.

I could probably adapt it to loop endlessly and simply exit once X-Next-Page either vanishes or equals the current page (haven't looked into what GitLab sends on the last page)... But I don't really need the script anymore, so probably won't bother.

Though as a bonus, mine is parallelised a bit; I was lucky and didn't need to worry about rate-limiting as I was only averaging around 300 calls a minute (out of the maximum of 600).

@Atarity

This comment has been minimized.

Copy link

@Atarity Atarity commented Apr 28, 2020

It is not removed artifacts from the 1st page of my pipelines list for some reason. It also missed .status attribute in console log. The rest is as advertised, thanks!

@philipptempel

This comment has been minimized.

Copy link

@philipptempel philipptempel commented Jul 21, 2020

It is not removed artifacts from the 1st page of my pipelines list for some reason. It also missed .status attribute in console log. The rest is as advertised, thanks!

@Atarity Check the source code and you will find the hint #starting with page 2 skipping most recent 100 Jobs thus it is intended that the first page of artifacts are not removed.

@voiski

This comment has been minimized.

Copy link

@voiski voiski commented Sep 23, 2020

The response can have a json with breaking lines \n. Consider removing it like ${response//\\n/}

response=${response//\\n/}
length=$(echo $response | jq '. | length')

Also, you can easy simulate next page checking [ $length -ne 0 ] and having the page loop to 1000 or more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.