Skip to content

Instantly share code, notes, and snippets.

@MetalArend
Last active June 16, 2024 10:11
Show Gist options
  • Save MetalArend/38db0f86f71a220d101f2a5ae49fc78c to your computer and use it in GitHub Desktop.
Save MetalArend/38db0f86f71a220d101f2a5ae49fc78c to your computer and use it in GitHub Desktop.
Run a GitLab Runner on your Swarm
version: '3.4'
secrets:
# Find your registration token at: "Your project" > "Settings" > "CI/CD" > "Runners settings" > "Specific Runners" (look for registration token)
# Register it as `GITLAB_REGISTRATION_TOKEN`: `docker secret create GITLAB_REGISTRATION_TOKEN YOUR_REGISTRATION_TOKEN`
GITLAB_REGISTRATION_TOKEN:
external: true
# Find your personal access token at: "Your user account" > "Settings" > "Access Tokens" > "Create personal access token" (for api)
# Register it as `GITLAB_PERSONAL_ACCESS_TOKEN`: `docker secret create GITLAB_PERSONAL_ACCESS_TOKEN <YOUR ACCESS TOKEN>`
GITLAB_PERSONAL_ACCESS_TOKEN:
external: true
services:
# Gitlab Runner - https://gitlab.com/gitlab-org/gitlab-runner
runner:
image: gitlab/gitlab-runner:latest
environment:
- CONCURRENT=8
- REGISTER_LOCKED=1
- REGISTER_NON_INTERACTIVE=1
- RUNNER_EXECUTOR=docker
- DOCKER_IMAGE=docker
- DOCKER_VOLUMES=/var/run/docker.sock:/var/run/docker.sock
- RUNNER_NAME=docker
- API_URL=https://gitlab.com/api/v4
- CI_SERVER_URL=https://gitlab.com/ci
entrypoint: "bash"
secrets:
- GITLAB_REGISTRATION_TOKEN
command: |
-c '
set -e
printf "Setting configuration...\\n"
export REGISTRATION_TOKEN="$$(cat /run/secrets/GITLAB_REGISTRATION_TOKEN)"
sed -i "s/^concurrent = .*/concurrent = $${CONCURRENT}/" /etc/gitlab-runner/config.toml
printf "\\n"
printf "Registering runner...\\n"
gitlab-runner register --non-interactive
printf "\\n"
printf "List runners...\\n"
gitlab-runner list
printf "\\n"
printf "Running runner...\\n"
gitlab-runner run --user=gitlab-runner --working-directory=/home/gitlab-runner --metrics-server=:9252
'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
deploy:
mode: global
placement:
constraints:
- node.role == manager
labels:
- "traefik.enable=false"
healthcheck:
test: ["CMD-SHELL", "gitlab-runner verify --name docker 2>&1 | grep --quiet \"is alive\""]
start_period: 10s
interval: 10s
timeout: 10s
retries: 10
# Gitlab Manager to unregister GitLab Runners
manager:
image: alpine:latest
environment:
- API_URL=https://gitlab.com/api/v4
- CI_SERVER_URL=https://gitlab.com/ci
secrets:
- GITLAB_PERSONAL_ACCESS_TOKEN
entrypoint: sh
command: |
-c '
set -e
printf "Installing dependencies...\\n"
apk --no-cache add curl jq
printf "\\n"
export PERSONAL_ACCESS_TOKEN="$$(cat /run/secrets/GITLAB_PERSONAL_ACCESS_TOKEN)"
while true; do
printf "Checking runners...\\n"
curl -sS --header "PRIVATE-TOKEN: $${PERSONAL_ACCESS_TOKEN}" "$${API_URL}/runners?per_page=100" | \
jq -c ".[] | select(false==.is_shared) | select(\"online\"==.status) | .id" | \
while read RUNNER_ID; do
printf "Runner $${RUNNER_ID} is online\\n"
done
curl -sS --header "PRIVATE-TOKEN: $${PERSONAL_ACCESS_TOKEN}" "$${API_URL}/runners?per_page=100" | \
jq -c ".[] | select(false==.is_shared) | select(\"online\"!=.status) | .id" | \
while read RUNNER_ID; do
printf "Deleting runner $${RUNNER_ID}...\\n"
curl -sS --request DELETE --header "PRIVATE-TOKEN: $${PERSONAL_ACCESS_TOKEN}" "$${API_URL}/runners/$${RUNNER_ID}"
done
printf "All offline runners deleted\\n"
printf "Waiting for 24 hours...\\n"
sleep 24h
done
printf "\\n"
'
deploy:
labels:
- "traefik.enable=false"
healthcheck:
test: ["CMD-SHELL", "command -v curl"]
start_period: 10s
interval: 10s
timeout: 10s
retries: 10
# Gitlab Runner Docker Cleanup - https://gitlab.com/gitlab-org/gitlab-runner-docker-cleanup
cleaner:
image: quay.io/gitlab/gitlab-runner-docker-cleanup
environment:
- CHECK_PATH=/data
- LOW_FREE_SPACE=10G
- EXPECTED_FREE_SPACE=20G
- LOW_FREE_FILES_COUNT=1048576
- EXPECTED_FREE_FILES_COUNT=2097152
- USE_DF=1
- CHECK_INTERVAL=10s
- RETRY_INTERVAL=30s
- DEFAULT_TTL=60m
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /data:/data
deploy:
restart_policy:
condition: any
labels:
- "traefik.enable=false"
@tamasgal
Copy link

tamasgal commented Nov 8, 2021

This does not work anymore, I get

root@pi1080:~/composes/km3net-gitlab-runners# docker service logs km3net-gitlab-runners_manager
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | Installing dependencies...
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | (1/7) Installing ca-certificates (20191127-r5)
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | (2/7) Installing brotli-libs (1.0.9-r5)
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | (3/7) Installing nghttp2-libs (1.43.0-r0)
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | (4/7) Installing libcurl (7.79.1-r0)
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | (5/7) Installing curl (7.79.1-r0)
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | (6/7) Installing oniguruma (6.9.7.1-r0)
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | (7/7) Installing jq (1.6-r1)
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | Executing busybox-1.33.1-r3.trigger
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | Executing ca-certificates-20191127-r5.trigger
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | OK: 9 MiB in 21 packages
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    |
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | Checking runners...
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | jq: error (at <stdin>:0): Cannot index string with string "is_shared"
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | jq: error (at <stdin>:0): Cannot index string with string "is_shared"
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | All offline runners deleted
km3net-gitlab-runners_manager.1.u14wqoara1ep@pi1080    | Waiting for 24 hours...

Is there maybe a newer version of this workflow?

@nixmomo
Copy link

nixmomo commented Dec 28, 2021

it is working as expected, the error message is shown if no runner is connected from this swarm stack

here is the log output

Installing dependencies...,
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/aarch64/APKINDEX.tar.gz,
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/aarch64/APKINDEX.tar.gz,
(1/7) Installing ca-certificates (20191127-r7),
(2/7) Installing brotli-libs (1.0.9-r5),
(3/7) Installing nghttp2-libs (1.46.0-r0),
(4/7) Installing libcurl (7.80.0-r0),
(5/7) Installing curl (7.80.0-r0),
(6/7) Installing oniguruma (6.9.7.1-r0),
(7/7) Installing jq (1.6-r1),
Executing busybox-1.34.1-r3.trigger,
Executing ca-certificates-20191127-r7.trigger,
OK: 9 MiB in 21 packages,
,
Checking runners...,
All offline runners deleted,
Waiting for 24 hours...

@tamasgal
Copy link

I could not get it working, the runner were showing up in GitLab and also assigned to jobs but they were simply hanging, but I'll give it another try! Thanks for confirming that it's still working.

@nixmomo
Copy link

nixmomo commented Jan 12, 2022

if it is not working please provide more informations and logs from container .... if it hangs without any errors in logs, it can be a network issue

@nixmomo
Copy link

nixmomo commented Jan 12, 2022

Ah one thing, the api has changed a bit, you need to edit the compose from
curl -sS --header "PRIVATE-TOKEN: $${PERSONAL_ACCESS_TOKEN}" "$${API_URL}/runners?per_page=100" | \
to
curl -sS --header "PRIVATE-TOKEN: $${PERSONAL_ACCESS_TOKEN}" "$${API_URL}/runners/all?per_page=100" | \
in line 93 and 88
You see the /all in the api call?
And if you use it for shared runners you need to update line 94 and 89 from
jq -c ".[] | select(false==.is_shared) | select(\"online\"!=.status) | .id" | \
to
jq -c ".[] | select(true==.is_shared) | select(\"online\"!=.status) | .id" | \

i made this changes a long time ago in my private repos but in general its for deleting runners only and should work without this changes

@FazelMan
Copy link

Thank You <3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment