Skip to content

Instantly share code, notes, and snippets.

@m0un10
Last active September 4, 2020 09:33
Show Gist options
  • Save m0un10/49a307028e852ab510a2dc14418412a5 to your computer and use it in GitHub Desktop.
Save m0un10/49a307028e852ab510a2dc14418412a5 to your computer and use it in GitHub Desktop.

GCP cached permissions issue

GCP has an issue which surfaces when service accounts are recreated with the same name but without the old policies being removed. It is confusing because the GUI and CLI will show that permissions are there and it will even let you re-add them BUT, anytime you try to do something that requires the permissions it won't work. For example, if you try to push an image it may say that you don't have storage.buckets.get even thought everything shows that you are part of storage.admin.

Reproducing the issue

Set the values to match your environment

export ACCOUNT=<service-account>
export PROJECT_ID=<project-id>
export IMAGE_NAME=gcr.io/$PROJECT_ID/your-image:your-tag

Create a service account with the required permissions and generate a key for it. Let's create a script called create.sh that does exactly that.

gcloud -q iam service-accounts create ${ACCOUNT} --display-name ${ACCOUNT}
gcloud -q projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:${ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com --role roles/storage.admin
gcloud iam service-accounts keys create key.json \
  --iam-account ${ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com
gcloud projects get-iam-policy ${PROJECT_ID}  \
--flatten="bindings[].members" \
--format='table(bindings.role)' \
--filter="bindings.members:${ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com"

When we run the script we will see the expected permissions

ROLE
roles/storage.admin

Now let's just check that an image can be pushed

docker login -u _json_key -p "$(cat key.json)" https://gcr.io
docker push $IMAGE_NAME

Cool. It worked. Ok, now let's delete the account and re-run create.sh

gcloud -q iam service-accounts delete ${ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com
./create.sh

That's odd, now when we re-run docker login -u _json_key -p "$(cat key.json)" https://gcr.io && docker push $IMAGE_NAME it is failing with the following issue.

denied: Token exchange failed for project '****'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control

But it shows that we have the right permissions in the GUI and in the CLI!!!

Fixing the issue

The issue is caused by the old permission hanging around. In fact, even if we re-create the service account and don't add any permissions, the old permissions show in the GUI and CLI. And if we try to add the permission, it will allow it but it won't actually be applied.

To fix, we have to make sure we delete the service and remove the permissions before we recreate it.

gcloud -q iam service-accounts delete ${ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com
gcloud -q projects remove-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:${ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com --role roles/storage.admin

After the above is ran we can re-run create.sh, wait for about 10 seconds and then try to login and push (i.e. docker login -u _json_key -p "$(cat key.json)" https://gcr.io && docker push $IMAGE_NAME), it will be successful. If not, wait a bit longer and do the push again, it will work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment