Skip to content

Instantly share code, notes, and snippets.

@swapnilgm
Last active August 4, 2020 08:38
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save swapnilgm/5c4d5506811e63c32ab3d73c4171d30f to your computer and use it in GitHub Desktop.
Save swapnilgm/5c4d5506811e63c32ab3d73c4171d30f to your computer and use it in GitHub Desktop.

Cloud storage comparison

Limits

Resource AWS ABS Swift GCS Alicloud
bucket name length 3-63 char resourcegrp 1-90char, storage-account 3-24char, container 3-63char 256 bytes 3-64 char, no goole prefix or mispell 3-63 char
object name unlimited 1-1024 1024 bytes 1024 1023
buckets per account default 100, on demand no limit 250 storage account, no limit on bucket per account - - 30 buckets per region
Objects per bucket unlimited capacity limits 2 PB for US and Europe, 500 TB for all other regions per stoarge account unlimited unlimited from forum unlimited
Object Read requests per bucket autoscale 20000req/sec per storage account - 5000 reads per second, autoscales as needed 10GBps
Objects Write request per bucket autoscale consider in above - 1000 reads per second, autoscales as needed 10GBps
Max Object size 5 TB 4.75 TB 5 GB 5 TB 5 GB, 48GB with multipart
Chunk size 5MB to 5 GB up to 100 MiB up to 5 GB - 5GB
Max no. of Chunks 10,000 50,000 1,000 32x
Writes per object once per second
Writes per bucket 20000 per second request per account no limits
HTTP Compression [No (https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) No yes

Azure Blob Storage (ABS)

Resource Limit
Number of storage accounts per region per subscription 2001
Max storage account capacity 500 TiB2
Max number of blob containers, blobs, file shares, tables, queues, entities, or messages per storage account No limit
Maximum request rate per storage account 20,000 requests per second2
Max ingress3 per storage account (US Regions) 10 Gbps if RA-GRS/GRS enabled, 20 Gbps for LRS/ZRS4
Max egress3 per storage account (US Regions) 20 Gbps if RA-GRS/GRS enabled, 30 Gbps for LRS/ZRS4
Max ingress3 per storage account (Non-US regions) 5 Gbps if RA-GRS/GRS enabled, 10 Gbps for LRS/ZRS4
Max egress3 per storage account (Non-US regions) 10 Gbps if RA-GRS/GRS enabled, 15 Gbps for LRS/ZRS4
Max size of single blob container Same as max storage account capacity
Max number of blocks in a block blob or append blob 50,000 blocks
Max size of a block in a block blob 100 MiB
Max size of a block blob 50,000 X 100 MiB (approx. 4.75 TiB)
Max size of a block in an append blob 4 MiB
Max size of an append blob 50,000 x 4 MiB (approx. 195 GiB)
Max size of a page blob 8 TiB
Max number of stored access policies per blob container 5
Target throughput for single blob Up to 60 MiB per second, or up to 500 requests per second

AWS S3

Item Specification
Maximum object size 5 TB
Maximum number of parts per upload 10,000
Part numbers 1 to 10,000 (inclusive)
Part size 5 MB to 5 GB, last part can be < 5 MB
Maximum number of parts returned for a list parts request 1000
Maximum number of multipart uploads returned in a list multipart uploads request 1000

References

  1. GCS: https://cloud.google.com/storage/quotas
  2. AWS: https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
  3. Azure ABS: https://docs.microsoft.com/en-us/azure/storage/common/storage-scalability-targets?toc=%2fazure%2fstorage%2fblobs%2ftoc.json
  4. Openstack Swift: https://docs.openstack.org/mitaka/config-reference/object-storage/features.html
  5. Alicloud OSS https://www.alibabacloud.com/help/doc-detail/54464.htm?spm=a2c63.p38356.b99.6.5f627d91oCAUf5

Volume Resizing

AWS-EBS GCE-PD Azure disks Cinder Alicloud
Resize supported from IaaS yes yes yes yes yes
Resize supported on k8s yes yes yes yes ?
Volume resize while attached to running instance yes yes ? No No

AWS-EBS

You can increase the volume size, change the volume type, or adjust the performance of your EBS volumes. If your instance supports Elastic Volumes, you can do so without detaching the volume or restarting the instance. This allows you to continue using your application while changes take effect.

Elastic Volumes are supported on the following instances:

  • All current-generation instances
  • Previous-generation instance families C1, C3, CC2, CR1, G2, I2, M1, M3, and R3

Monitoring

  • When you modify an EBS volume, it goes through a sequence of states. The volume enters the modifying state, the optimizing state, and finally the completed state. At this point, the volume is ready to be further modified.
  • Volume modification changes take effect as follows:
    • Size changes usually take a few seconds to complete and take effect after a volume is in the Optimizing state.

    • Performance (IOPS) changes can take from a few minutes to a few hours to complete and are dependent on the configuration change being made.

    • It may take up to 24 hours for a new configuration to take effect, and in some cases more, such as when the volume has not been fully initialized. Typically, a fully used 1-TiB volume takes about 6 hours to migrate to a new performance configuration.

Limitation

  • After provisioning over 32,000 IOPS on an existing io1 volume, you may need to do one of the following to see the full performance improvements:
    • Detach and attach the volume.
    • Restart the instance.
  • Decreasing the size of an EBS volume is not supported. However, you can create a smaller volume and then migrate your data to it using an application-level tool such as rsync.
  • ⚠️ While m3.medium instances fully support volume modification, m3.large, m3.xlarge, and m3.2xlarge instances might not support all volume modification features.

GCE-PD

  • You can resize disks at any time, regardless of whether the disk is attached to a running instance.
  • After you resize your zonal persistent disk, you must configure the file system on the disk to use the additional disk space.

Azure disk

  • The new size should be greater than the existing disk size.

Kubernetes volume resize

  • Beta since kubernetes 1.11
  • Edit the storage size in PersitentVolumeClaim.
  • Required enabling the feature gate, ExpandPersistentVolumes, as well as the admission controller, PersistentVolumeClaimResize. By default enabled in 1.11+.
  • Current support from AWS-EBS, GCE-PD, Azure Disk, Azure File, Glusterfs, Cinder, Portworx, and Ceph RBD.
  • Enbale it by setting allowVolumeExpansion field to true in PVC configured StorageClass object.
  • Block storage volume types such as GCE-PD, AWS-EBS, Azure Disk, Cinder, and Ceph RBD typically require a file system expansion before the additional space of an expanded volume is usable by pods. Kubernetes takes care of this automatically whenever the pod(s) referencing your volume are restarted.
  • Post volume resize, if the PersistentVolumeClaim has the status FileSystemResizePending, it is safe to recreate the pod using the PersistentVolumeClaim.

Alicloud

  • Resize the data disks that are attached to an instance only when the instance is in the Running or Stopped status.
  • You must restart the instance in the ECS console to apply the changes. This action causes your instance to stop working and may cause your business to be interrupted, so please proceed with caution.

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment