I had a bit of trouble trying to configure permissions to upload files from my Google Compute Engine instance to my Google Cloud Storage bucket. The process isn't as intuitive as you think. There are a few permissions issues that need to be configured before this can happen. Here are the steps I took to get things working.
Let's say you want to upload yourfile.txt
to a GCS bucket from your virtual machine.
You can use the gsutil
command line tool that comes installed on all GCE instances.
If you've never used the gcloud
or gsutil
command line tools on this machine before, you will need to initialize them with a service account.
On the GCE instance run the following to set up:
gcloud init
The setup will ask you to choose the account you would like to use to perform operations for this configuration, and give you two options:
- 1234567890-compute@developer.gserviceaccount.com
- Log in with a new account
Choose number 1 to use a service account. If this is a shared machine and you log in with your personal account, your credentials could be used by anyone else on the machine.
Once logged in, try uploading your file with the gsutil
command, which might look like this:
gsutil cp /home/you/yourfile.txt gs://your-bucket
You will notice you're faced with a AccessDeniedException: 403 Insufficient Permission
message, and despite your best efforts it's difficult to debug.
Next, you will need to enable API access for the virtual machine. By default your machine should have read access to the buckets in the same project, but configuration is required before you can write to them.
Navigate to console.cloud.google.com and select your project from the drop down menu. Next, select your virtual machine and click STOP
in the top menu bar.
Once your virtual machine has been stopped, click on it's name and then EDIT
in the top menu bar.
Scroll down until you see a header called Access Scopes
, which will likely be on the Allow default access
selection. Select Set access for each API
as your option, then scroll down until you see Storage
, which is likely set on READ
: change it to READ WRITE
, or whatever you feel is necessary for your use case.
Save your changes and restart your virtual machine.
Once you've restarted your virtual machine, try to upload the file again:
gsutil cp /home/you/yourfile.txt gs://your-bucket
If you encounter the AccessDeniedException: 403 Insufficient Permission
message again, navigate to your current home directory, and remove the .gsutil cache folder.
rm -r ~/.gsutil
You should now be able to upload successfully to your storage bucket.
gsutil cp /home/you/yourfile.txt gs://your-bucket
...
Operation completed over 1 objects.
(This section shouldn't be necessary - I am able to upload without explicit permissions set on the account in IAM)
If the above doesn't work, you may need to enable additional IAM permissions for the service account.
First, find your service account with the following command
gcloud config list account
...
1234567890-compute@developer.gserviceaccount.com
Navigate to console.cloud.google.com and select your project from the drop down menu.
Find your service account in the members list, and click the edit pencil on the right hand side. Then add any permissions as needed.
thanks a lot, it's really helpful for my instances.