|from boto3.session import Session|
|from botocore.client import Config|
|from botocore.handlers import set_list_objects_encoding_type_url|
|ACCESS_KEY = "xx"|
|SECRET_KEY = "yy"|
|session = Session(aws_access_key_id=ACCESS_KEY,|
|s3 = session.resource('s3', endpoint_url='https://storage.googleapis.com',|
|bucket = s3.Bucket('yourbucket')|
|for f in bucket.objects.all():|
Google Cloud implements the latest s3 protocol. Most errors while trying to make it work were like the following:
Which meant (in this order): Forgot to set region, mistyped google's region, have not set the proper protocol version and having boto append "encoding=url" as query string which Google Storage won't accept. The last one was tricky to unregister.
Setting boto logger helped track, as it helped reading the header dump (sometimes it would complains about an amz-sha256 header). Thanks to all github issues and pieces around I managed to make it work. Put it all together in this gist so other people can get over this and get work done.
Thanks for sharing! This also works from AWS lambda's, where boto3 is pre-installed :)
Took me a while to figure out what access key and secret to use, but here's what you need to do to get them: https://cloud.google.com/storage/docs/migrating#keys
Sorry folks, github doesn't makes it easy to track gist comments. This is working to me to move an AWS project to GCP. I should have mentioned that you need to create the storage instance in "interoperability mode" to get the AWS like credentials. I had to score the docs and boto code to figure that out, so I'm glad that helped.
Thank you, that helped a lot.