Skip to content

Instantly share code, notes, and snippets.

@ryderdamen
Created December 4, 2018 15:43
Show Gist options
  • Save ryderdamen/926518ddddd46dd4c8c2e4ef5167243d to your computer and use it in GitHub Desktop.
Save ryderdamen/926518ddddd46dd4c8c2e4ef5167243d to your computer and use it in GitHub Desktop.
Uploading Files from Google Compute Engine (GCE) VMs to Google Cloud Storage (GCS)

Uploading Files from Google Compute Engine (GCE) VMs to Google Cloud Storage (GCS)

I had a bit of trouble trying to configure permissions to upload files from my Google Compute Engine instance to my Google Cloud Storage bucket. The process isn't as intuitive as you think. There are a few permissions issues that need to be configured before this can happen. Here are the steps I took to get things working.

Let's say you want to upload yourfile.txt to a GCS bucket from your virtual machine. You can use the gsutil command line tool that comes installed on all GCE instances.

If you've never used the gcloud or gsutil command line tools on this machine before, you will need to initialize them with a service account.

Set Up Your Service Account

On the GCE instance run the following to set up:

gcloud init

The setup will ask you to choose the account you would like to use to perform operations for this configuration, and give you two options:

  1. 1234567890-compute@developer.gserviceaccount.com
  2. Log in with a new account

Choose number 1 to use a service account. If this is a shared machine and you log in with your personal account, your credentials could be used by anyone else on the machine.

Once logged in, try uploading your file with the gsutil command, which might look like this:

gsutil cp /home/you/yourfile.txt gs://your-bucket

You will notice you're faced with a AccessDeniedException: 403 Insufficient Permission message, and despite your best efforts it's difficult to debug.

Enabling API Access for the GCE Instance

Next, you will need to enable API access for the virtual machine. By default your machine should have read access to the buckets in the same project, but configuration is required before you can write to them.

Navigate to console.cloud.google.com and select your project from the drop down menu. Next, select your virtual machine and click STOP in the top menu bar.

Once your virtual machine has been stopped, click on it's name and then EDIT in the top menu bar.

Scroll down until you see a header called Access Scopes, which will likely be on the Allow default access selection. Select Set access for each API as your option, then scroll down until you see Storage, which is likely set on READ: change it to READ WRITE, or whatever you feel is necessary for your use case.

Save your changes and restart your virtual machine.

Removing gsutil Cache

Once you've restarted your virtual machine, try to upload the file again:

gsutil cp /home/you/yourfile.txt gs://your-bucket

If you encounter the AccessDeniedException: 403 Insufficient Permission message again, navigate to your current home directory, and remove the .gsutil cache folder.

rm -r ~/.gsutil

You should now be able to upload successfully to your storage bucket.

gsutil cp /home/you/yourfile.txt gs://your-bucket
...
Operation completed over 1 objects.

Enabling IAM Permissions

(This section shouldn't be necessary - I am able to upload without explicit permissions set on the account in IAM)

If the above doesn't work, you may need to enable additional IAM permissions for the service account.

First, find your service account with the following command

gcloud config list account
...
1234567890-compute@developer.gserviceaccount.com

Navigate to console.cloud.google.com and select your project from the drop down menu.

Find your service account in the members list, and click the edit pencil on the right hand side. Then add any permissions as needed.

@gchandn3413
Copy link

This is so helpful. Thanks a lot!!

@lpmartineau
Copy link

Thank you. I was struggling with this and your comment on deleting the .gsutil folder solved it.

@Keerthihegde
Copy link

Keerthihegde commented Aug 15, 2019

Thanks a lot for this!! rm -r ~/.gsutil was really helpful.

@Deleplace
Copy link

Wow. Thank you so much. This was all very non-intuitive to me. Glad I found your nice page.

@AK-mocha
Copy link

Thanks a million. It is really helpful.

@Triano123
Copy link

thanks a lot, it's really helpful for my instances.

@vrobert78
Copy link

On a Windows machine, the .gsutil folder is in C:\Users\yourname.gsutil

@MikeOfZen
Copy link

deleting the .gsutil did it!

@prabowomurti
Copy link

Thank you : "Enabling API Access for the GCE Instance" solved my issue.

@eustin
Copy link

eustin commented Feb 29, 2020

This step was the key for me:

rm -r ~/.gsutil

Thank you! You're the best.

@prats0599
Copy link

THANK YOUU SO MUCH!!

@vyolla
Copy link

vyolla commented Mar 26, 2020

GREAT!

@bitmanlger
Copy link

YES, thank you! rm -r ~/.gsutil soo easy to miss!

@NageshJ2014
Copy link

Pretty good stuff, After adding the required API access, key is to remove the .gsutil cache folder.
Nice one , Thanks

@theArina
Copy link

thank you so much !

@jason9075
Copy link

Thanks !!

@maurorappa
Copy link

thanks, please mention also the option '-m' for parallel uploads : https://cloud.google.com/storage/docs/gsutil/commands/cp

@nikhil-salodkar
Copy link

Thank you so much

@DolanP
Copy link

DolanP commented Jan 18, 2021

Thank you!!!

@skaiser
Copy link

skaiser commented Jan 21, 2021

Thanks!!!

@Trumeet
Copy link

Trumeet commented Mar 4, 2021

Your rm -rf saved my life as the permission error is really hard to debug. (like for my situation both cp and del works but setmeta not. After deleting the cache it works magically.)

@gozi1123
Copy link

Thanks a lot!

@quoccuongmmt
Copy link

thanks ....!!

@AlexandrosChrtn
Copy link

Very detailed guide. Thanks!

@s1846118
Copy link

Thanks for this. Really helpful.

@nandoverona
Copy link

Tks! Very helpful.
Another option is to create the Compute Engine with an open scope, allowing any API call.

--scopes=https://www.googleapis.com/auth/cloud-platform \

@cajiva
Copy link

cajiva commented Jun 9, 2022

Thank you!!! worked out perfectly.

@Ban921
Copy link

Ban921 commented Sep 14, 2023

Thank you!!! worked out perfectly.
And it saves me a lot of time.

@Mukesh-yadav-AI
Copy link

The above solution doesn't work for me getting access denied error

@lorenzoritter
Copy link

lorenzoritter commented Nov 4, 2023

Thanks for the advice to remove the cache. It resolved the issue for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment