Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Uploading Files from Google Compute Engine (GCE) VMs to Google Cloud Storage (GCS)

Uploading Files from Google Compute Engine (GCE) VMs to Google Cloud Storage (GCS)

I had a bit of trouble trying to configure permissions to upload files from my Google Compute Engine instance to my Google Cloud Storage bucket. The process isn't as intuitive as you think. There are a few permissions issues that need to be configured before this can happen. Here are the steps I took to get things working.

Let's say you want to upload yourfile.txt to a GCS bucket from your virtual machine. You can use the gsutil command line tool that comes installed on all GCE instances.

If you've never used the gcloud or gsutil command line tools on this machine before, you will need to initialize them with a service account.

Set Up Your Service Account

On the GCE instance run the following to set up:

gcloud init

The setup will ask you to choose the account you would like to use to perform operations for this configuration, and give you two options:

  1. 1234567890-compute@developer.gserviceaccount.com
  2. Log in with a new account

Choose number 1 to use a service account. If this is a shared machine and you log in with your personal account, your credentials could be used by anyone else on the machine.

Once logged in, try uploading your file with the gsutil command, which might look like this:

gsutil cp /home/you/yourfile.txt gs://your-bucket

You will notice you're faced with a AccessDeniedException: 403 Insufficient Permission message, and despite your best efforts it's difficult to debug.

Enabling API Access for the GCE Instance

Next, you will need to enable API access for the virtual machine. By default your machine should have read access to the buckets in the same project, but configuration is required before you can write to them.

Navigate to console.cloud.google.com and select your project from the drop down menu. Next, select your virtual machine and click STOP in the top menu bar.

Once your virtual machine has been stopped, click on it's name and then EDIT in the top menu bar.

Scroll down until you see a header called Access Scopes, which will likely be on the Allow default access selection. Select Set access for each API as your option, then scroll down until you see Storage, which is likely set on READ: change it to READ WRITE, or whatever you feel is necessary for your use case.

Save your changes and restart your virtual machine.

Removing gsutil Cache

Once you've restarted your virtual machine, try to upload the file again:

gsutil cp /home/you/yourfile.txt gs://your-bucket

If you encounter the AccessDeniedException: 403 Insufficient Permission message again, navigate to your current home directory, and remove the .gsutil cache folder.

rm -r ~/.gsutil

You should now be able to upload successfully to your storage bucket.

gsutil cp /home/you/yourfile.txt gs://your-bucket
...
Operation completed over 1 objects.

Enabling IAM Permissions

(This section shouldn't be necessary - I am able to upload without explicit permissions set on the account in IAM)

If the above doesn't work, you may need to enable additional IAM permissions for the service account.

First, find your service account with the following command

gcloud config list account
...
1234567890-compute@developer.gserviceaccount.com

Navigate to console.cloud.google.com and select your project from the drop down menu.

Find your service account in the members list, and click the edit pencil on the right hand side. Then add any permissions as needed.

@dupski

This comment has been minimized.

Copy link

@dupski dupski commented Mar 29, 2019

Thanks so much for this!! Had the same frustration. Removing the cache solved my problem - seems like a bit of a bug in the platform to me for a fairly fundamental need!

Out of interest I didn't need to do the gcloud init step (Ubuntu 18.04 image), but I did have to stop the VM and change the Cloud Storage API scope to Read/Write instead of ReadOnly, and I'm guessing that is what was cached.

@gchandn3413

This comment has been minimized.

Copy link

@gchandn3413 gchandn3413 commented Apr 7, 2019

This is so helpful. Thanks a lot!!

@lpmartineau

This comment has been minimized.

Copy link

@lpmartineau lpmartineau commented Apr 17, 2019

Thank you. I was struggling with this and your comment on deleting the .gsutil folder solved it.

@Keerthihegde

This comment has been minimized.

Copy link

@Keerthihegde Keerthihegde commented Aug 15, 2019

Thanks a lot for this!! rm -r ~/.gsutil was really helpful.

@Deleplace

This comment has been minimized.

Copy link

@Deleplace Deleplace commented Aug 22, 2019

Wow. Thank you so much. This was all very non-intuitive to me. Glad I found your nice page.

@AK-mocha

This comment has been minimized.

Copy link

@AK-mocha AK-mocha commented Aug 24, 2019

Thanks a million. It is really helpful.

@Triano123

This comment has been minimized.

Copy link

@Triano123 Triano123 commented Sep 2, 2019

thanks a lot, it's really helpful for my instances.

@vrobert78

This comment has been minimized.

Copy link

@vrobert78 vrobert78 commented Oct 1, 2019

On a Windows machine, the .gsutil folder is in C:\Users\yourname.gsutil

@MikeOfZen

This comment has been minimized.

Copy link

@MikeOfZen MikeOfZen commented Nov 26, 2019

deleting the .gsutil did it!

@prabowomurti

This comment has been minimized.

Copy link

@prabowomurti prabowomurti commented Feb 26, 2020

Thank you : "Enabling API Access for the GCE Instance" solved my issue.

@eustin

This comment has been minimized.

Copy link

@eustin eustin commented Feb 29, 2020

This step was the key for me:

rm -r ~/.gsutil

Thank you! You're the best.

@prats0599

This comment has been minimized.

Copy link

@prats0599 prats0599 commented Mar 6, 2020

THANK YOUU SO MUCH!!

@vyolla

This comment has been minimized.

Copy link

@vyolla vyolla commented Mar 26, 2020

GREAT!

@bitmanlger

This comment has been minimized.

Copy link

@bitmanlger bitmanlger commented May 28, 2020

YES, thank you! rm -r ~/.gsutil soo easy to miss!

@NageshJ2014

This comment has been minimized.

Copy link

@NageshJ2014 NageshJ2014 commented Jun 14, 2020

Pretty good stuff, After adding the required API access, key is to remove the .gsutil cache folder.
Nice one , Thanks

@theArina

This comment has been minimized.

Copy link

@theArina theArina commented Jul 30, 2020

thank you so much !

@jason9075

This comment has been minimized.

Copy link

@jason9075 jason9075 commented Oct 31, 2020

Thanks !!

@maurorappa

This comment has been minimized.

Copy link

@maurorappa maurorappa commented Nov 12, 2020

thanks, please mention also the option '-m' for parallel uploads : https://cloud.google.com/storage/docs/gsutil/commands/cp

@nikhil-salodkar

This comment has been minimized.

Copy link

@nikhil-salodkar nikhil-salodkar commented Jan 5, 2021

Thank you so much

@DolanP

This comment has been minimized.

Copy link

@DolanP DolanP commented Jan 18, 2021

Thank you!!!

@skaiser

This comment has been minimized.

Copy link

@skaiser skaiser commented Jan 21, 2021

Thanks!!!

@Trumeet

This comment has been minimized.

Copy link

@Trumeet Trumeet commented Mar 4, 2021

Your rm -rf saved my life as the permission error is really hard to debug. (like for my situation both cp and del works but setmeta not. After deleting the cache it works magically.)

@gozi1123

This comment has been minimized.

Copy link

@gozi1123 gozi1123 commented May 19, 2021

Thanks a lot!

@quoccuongmmt

This comment has been minimized.

Copy link

@quoccuongmmt quoccuongmmt commented Jun 15, 2021

thanks ....!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment