How to use boto3 with google cloud storage and python to emulate s3 access.

list_objects_google_storage_boto3.py

from boto3.session import Session
from botocore.client import Config
from botocore.handlers import set_list_objects_encoding_type_url

import boto3

ACCESS_KEY = "xx"
SECRET_KEY = "yy"
boto3.set_stream_logger('')

session = Session(aws_access_key_id=ACCESS_KEY,
                  aws_secret_access_key=SECRET_KEY,
                  region_name="US-CENTRAL1")

session.events.unregister('before-parameter-build.s3.ListObjects',
                          set_list_objects_encoding_type_url)

s3 = session.resource('s3', endpoint_url='https://storage.googleapis.com',
                      config=Config(signature_version='s3v4'))


bucket = s3.Bucket('yourbucket')

for f in bucket.objects.all():
        print(f.key)

Never mind it works now.

Sorry folks, github doesn't makes it easy to track gist comments. This is working to me to move an AWS project to GCP. I should have mentioned that you need to create the storage instance in "interoperability mode" to get the AWS like credentials. I had to score the docs and boto code to figure that out, so I'm glad that helped.

Thank you, that helped a lot.
As for the generation of pre-signed URLs, just replace the 'AWSAccessKeyId' query param in the generated URL by 'GoogleAccessId' to make it work.
Neither boto3 nor botocore mention the literal 'GoogleAccessId' in their code, so you have to replace it by hand as follows:

url = s3.meta.client.generate_presigned_url(
    ClientMethod='get_object',
    Params={
        'Bucket': 'yourbucket',
        'Key': 'object.txt'
    }
)

url = url.replace('AWSAccessKeyId', 'GoogleAccessId')

Thanks for a minimum viable solution to the gcp interop issue. (Linking because I didn't understand your solution until reading the issue thread).

If you are wondering how to generate the ACCESS KEY and SECRET KEY: https://cloud.google.com/storage/docs/authentication/managing-hmackeys

Has anyone managed to get this working with a ResponseContentDisposition header? I'm receiving SignatureDoesNotMatch errors

Edit: Check my comment under(https://gist.github.com/gleicon/2b8acb9f9c0f22753eaac227ff997b34?permalink_comment_id=4721658#gistcomment-4721658) for solution

Try enabling boto3 logging. I'm not sure Google Object Storage support all headers or the same syntax (see the AwsAccessKeyId issue above). Also check if the signature version still right for the type of storage you've created.

I managed to get it working. The comment under https://stackoverflow.com/a/21028609/9553927 helped:

The signed urls worked for me. Although, I've tried to call generate_url() with the parameter response_headers and the value response-content-disposition but I got malformed signed urls. So my solution has been to concatenate '&response-content-disposition=attachment%3B%20filename%3D"{}"'.format(file_name) to the signed url and it worked.

params = {
    "Bucket": "xyz",
    "Key": blob_name,
}
ten_minutes = 600  # seconds
url = self.s3_client.generate_presigned_url(
    "get_object", Params=params, ExpiresIn=ten_minutes
).replace("AWSAccessKeyId", "GoogleAccessId")
url += '&response-content-disposition=attachment;filename="newFileName"'

Awesome, thanks for sharing !