Uploading Files from Google Compute Engine (GCE) VMs to Google Cloud Storage (GCS)

gce-to-gcs-uploads.md

Uploading Files from Google Compute Engine (GCE) VMs to Google Cloud Storage (GCS)

I had a bit of trouble trying to configure permissions to upload files from my Google Compute Engine instance to my Google Cloud Storage bucket. The process isn't as intuitive as you think. There are a few permissions issues that need to be configured before this can happen. Here are the steps I took to get things working.

Let's say you want to upload yourfile.txt to a GCS bucket from your virtual machine. You can use the gsutil command line tool that comes installed on all GCE instances.

If you've never used the gcloud or gsutil command line tools on this machine before, you will need to initialize them with a service account.

Set Up Your Service Account

On the GCE instance run the following to set up:

gcloud init

The setup will ask you to choose the account you would like to use to perform operations for this configuration, and give you two options:

1. 1234567890-compute@developer.gserviceaccount.com
2. Log in with a new account

Choose number 1 to use a service account. If this is a shared machine and you log in with your personal account, your credentials could be used by anyone else on the machine.

Once logged in, try uploading your file with the gsutil command, which might look like this:

gsutil cp /home/you/yourfile.txt gs://your-bucket

You will notice you're faced with a AccessDeniedException: 403 Insufficient Permission message, and despite your best efforts it's difficult to debug.

Enabling API Access for the GCE Instance

Next, you will need to enable API access for the virtual machine. By default your machine should have read access to the buckets in the same project, but configuration is required before you can write to them.

Navigate to console.cloud.google.com and select your project from the drop down menu. Next, select your virtual machine and click STOP in the top menu bar.

Once your virtual machine has been stopped, click on it's name and then EDIT in the top menu bar.

Scroll down until you see a header called Access Scopes, which will likely be on the Allow default access selection. Select Set access for each API as your option, then scroll down until you see Storage, which is likely set on READ: change it to READ WRITE, or whatever you feel is necessary for your use case.

Save your changes and restart your virtual machine.

Removing gsutil Cache

Once you've restarted your virtual machine, try to upload the file again:

gsutil cp /home/you/yourfile.txt gs://your-bucket

If you encounter the AccessDeniedException: 403 Insufficient Permission message again, navigate to your current home directory, and remove the .gsutil cache folder.

rm -r ~/.gsutil

You should now be able to upload successfully to your storage bucket.

gsutil cp /home/you/yourfile.txt gs://your-bucket
...
Operation completed over 1 objects.

Enabling IAM Permissions

(This section shouldn't be necessary - I am able to upload without explicit permissions set on the account in IAM)

If the above doesn't work, you may need to enable additional IAM permissions for the service account.

First, find your service account with the following command

gcloud config list account

...
1234567890-compute@developer.gserviceaccount.com

Navigate to console.cloud.google.com and select your project from the drop down menu.

Find your service account in the members list, and click the edit pencil on the right hand side. Then add any permissions as needed.

This is so helpful. Thanks a lot!!

Thank you. I was struggling with this and your comment on deleting the .gsutil folder solved it.

Thanks a lot for this!! rm -r ~/.gsutil was really helpful.

Wow. Thank you so much. This was all very non-intuitive to me. Glad I found your nice page.

Thanks a million. It is really helpful.

thanks a lot, it's really helpful for my instances.

On a Windows machine, the .gsutil folder is in C:\Users\yourname.gsutil

deleting the .gsutil did it!

Thank you : "Enabling API Access for the GCE Instance" solved my issue.

This step was the key for me:

rm -r ~/.gsutil

Thank you! You're the best.

THANK YOUU SO MUCH!!

GREAT!

YES, thank you! rm -r ~/.gsutil soo easy to miss!

Pretty good stuff, After adding the required API access, key is to remove the .gsutil cache folder.
Nice one , Thanks

thank you so much !

Thanks !!

thanks, please mention also the option '-m' for parallel uploads : https://cloud.google.com/storage/docs/gsutil/commands/cp

Thank you so much

Thank you!!!

Thanks!!!

Your rm -rf saved my life as the permission error is really hard to debug. (like for my situation both cp and del works but setmeta not. After deleting the cache it works magically.)

Thanks a lot!

thanks ....!!

Very detailed guide. Thanks!

Thanks for this. Really helpful.

Tks! Very helpful.
Another option is to create the Compute Engine with an open scope, allowing any API call.

--scopes=https://www.googleapis.com/auth/cloud-platform \

Thank you!!! worked out perfectly.

Thank you!!! worked out perfectly.
And it saves me a lot of time.

The above solution doesn't work for me getting access denied error

Thanks for the advice to remove the cache. It resolved the issue for me.