Skip to content

Instantly share code, notes, and snippets.

@harshavardhana
Last active October 11, 2022 22:09
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save harshavardhana/5d4d7410afb02ec9837e2b4d82a932e7 to your computer and use it in GitHub Desktop.
Save harshavardhana/5d4d7410afb02ec9837e2b4d82a932e7 to your computer and use it in GitHub Desktop.
API AWS S3 MinIO GCS (S3 compatibility)
ListObjectVersions ✔️ ✔️

GCS fails in this manner while they succeed on AWS S3 and MinIO fine.

~ mc ls --versions gcs/harshavardhana
mc: <ERROR> Unable to list folder. unrecognized option:Marker
API AWS S3 MinIO GCS (S3 compatibility)
GetBucketEncryption ✔️ ✔️

GCS fails in this manner while they succeed on AWS S3 and MinIO fine.

~  mc encrypt info gcs/harshavardhana
mc: <ERROR> Unable to get encryption info: 503 Service Unavailable.
API AWS S3 MinIO GCS (S3 compatibility)
PutBucketLifecycle ✔️ ✔️

GCS fails in this manner while they succeed on AWS S3 and MinIO fine.

~ mc ilm add gcs/harshavardhana --expiry-days 365
mc: <ERROR> Unable to set new lifecycle rules. The XML you provided was not well-formed or did not validate against our published schema.
API AWS S3 MinIO GCS (S3 compatibility)
SelectObjectContent ✔️ ✔️

GCS fails in this manner while they succeed on AWS S3 and MinIO fine.

~ mc sql gcs/harshavardhana/hosts.csv
mc: <ERROR> Unable to run sql Invalid argument.
API AWS S3 MinIO GCS (S3 compatibility)
MultiObjectDelete ✔️ ✔️

GCS fails in this manner while they succeed on AWS S3 and MinIO fine.

~ mc rm gcs/harshavardhana/hosts
Removing `gcs/harshavardhana/hosts`.
mc: <ERROR> Failed to remove `gcs/harshavardhana/hosts`. Invalid argument.
API AWS S3 MinIO GCS (S3 compatibility)
PutObjectTags ✔️ ✔️

GCS fails in this manner while they succeed on AWS S3 and MinIO fine.

~ mc tag set gcs/harshavardhana/hosts "key1=value1&key2=value2&key3=value3"
mc: <ERROR> Failed to set tags for https://storage.googleapis.com/harshavardhana/hosts: 503 Service Unavailable.

Summary

AWS S3 API Compatibility is quite limited, since few basic APIs do not work well. Other APIs that are not mentioned are not part of GCS S3 compatibility layer implementation are

@cboettig
Copy link

This is a great list. In addition to the above, it also appears that HTML-encoded paths are not correctly implemented on GCS(?) This seems not to break ls, find, etc but breaks most other operations on objects (cp, stat, etc)

minio/minio#15841 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment