aws-s3-notes.md

EC2 - Elastic Cloud Compute (Virtual Machines aka Compute Instances)

IAM - Identity and Access Management used for creating and managing users and groups.

S3 - Simple Storage Service. Object-based storage. Universal namespace.

NAS - Network Attached Storage.

ARN - Amazon Resource Name.

S3 Data Consitency Model

• Read after Write consistency for PUTS of new Objects.
• Eventual Consistency for overwrite PUTS and DELETES.

S3 Objects Attributes

• Key (name - lexicographically sorted)
• Value (data)
• Version ID
• Metadata (data of the data)
• Subresources
  • Access Control Lists (ACL)
  • Torrent

S3 Basics

• 99.99% availability
• 99.999999999% (11x9) durability
• Tiered Storage Available
• Lifecycle Management
• Versioning
• Encryption
• Secure your data using Access Control Lists and Bucket Policies

S3 Storage Tiers

• S3 - 99.99% availability, 99.999999999% durability, stored redundantly in 2 locations
• S3 IA (Infrequently Accessed)
• Reduced Redundancy Storage - 99.99% durability, stored in 1 location
• Glacier - cheap but for archival only and takes 3-5 hours to restore

S3 Charges

• Storage
• Requests
• Storage Management Pricing
• Data Transfer Pricing
• Transfer Acceleration

S3 Other Notes

• Object-based storage only.
• Not suitable for installing OS.
• HTTP 200 after successfull upload
• Buckets are a universal name space.
• Encryption
  • Client Side Encryption
  • Server Side Encryption
    • Amazon S3 Managed Keys (SSE-S3)
    • KMS (SSE-KMS)
    • Customer Provided Keys
• Control access to buckets using either a bucket ACL or using Bucket Policies
• By default buckets are private and all objects stored inside them are pivate.

S3 Versioning

• Stores all versions of an object
• Great backup tool
• Versioning cannot be disabled after enabling
• Integrates with Lifecycle rules
• MFA delete capability

S3 Cross Region Replication

• In order to work, versioning should be enabled. Replication region should be in a different region.
• Nav: Amazon S3 -> <bucket_name> -> 'Management' tab -> 'Replication' tab
• Versioning must be enabled on both source and destination buckets.
• Regions must be unique.
• Files in an existing bucket are not replicated automatically. All subsequent updated files will be replicated automatically.
• You cannot replicate to multiple buckets or use daisy chaining (at this time).
• Delete markers are replicated.
• Deleting individual versions or delete markers will not be replicated.
• Understand what Cross Region Replication is at a high level.

AWS CLI Commands

• aws configure
• aws s3 cp --recursive s3://abantej-s3-04-2018 s3://abantej-s3-04-2018-sydney

S3 Lifecycle Management

• Can be used in conjunction with versioning.
• Can be applied to current versions and previous versions.
• Following actions can now be done:
  • Transition to Standard - Infrequent Access Storage CLass (128kb and 30 days after the creation date.)
  • Permanently Delete

Content Delivery Network (CDN)

• A system of distributed servers that deliver webpages/webcontent to a user based on:
  • Geographic locations of the user
  • Origin of the webpage
  • Content delivery server

Edge Location

• The location where content will be cached. This is separate to an AWS Region/AZ.
• Edge Locations are not just READ only, you can write to them too.
• Objects are cached for the life of the TTL (Time to Live)
• You can clear cached objects, but you will be charged.

Origin

• The origin of all the files that the CDN will distribute. Can be:
  • S3 Bucket
  • EC2 Instance
  • Elastic Load Balancer
  • Route53

Distribution

• A CDN which consists of a collection of Edge Locations

CloudFront

• CloudFront can be used to deliver your entire website, including dynamic, static, streaming, and interactive content using a global network of edge locations.
• Requests for you content are automatically routed to the nearest edge location, so content is delivered with the best possible performance.
• CloudFront is optimized to work with other Amazon Web Services like:
  • S3
  • EC2
  • Elastic Load Balancing
  • Route 53
• CloudFront also works seamlessly with any non-AWS origin server, which stores the original, definitive versions of your files.

Web Distribution

• Typically used for Websites

RTMP

• Used for Media Streaming

S3 Security & Encryption

• By default, all newly created buckets are PRIVATE
• You can setup access control to your buckets using:
  • Bucket Policies
  • Access Control Lists
• S3 buckets can be configured to create access logs to which log all requests made to the S3 bucket. This can be done to another bucket.

S3 Encryption

• In transit
  • SSL/TLS
• At Rest
  • Server Side Encryption
    • S3 Managed Keys - SSE-S3
    • AWS Key Management Service, Managed Keys - SSE-KMS
    • Server Side Encryption With Customer Provided Keys - SSE-C
  • Client Side Encryption

AWS Storage Gateway

• A service that connects an on-premises software appliance with cloud-based storage to provide seamless and secure integration between an organization's on-premises IT environment and AWS's storage infrastructure. The service enables you to securely store data to the AWS cloud for scalable and cost-effective storage.
• The service is available for download as a virtual machine (VM) image that you install on a host in your datacenter.
• Storage supports either VMware ESXi or Microsoft Hyper-V.
• Once you've installed your gateway and associated it with your AWS account through the activation process, you can use the AWS management console to create the storage gateway option that is right for you.

Types of Storage Gateways

• File Gateway (NFS)
• Volumes Gateway (iSCSI)
  • Stored Volumes
  • Cached Volumes
• Tape Gateway (VTL)

File Gateway

• Files are stored as objects in your S3 buckets, accessed through a Network File System (NFS) mount point. Ownership, permissions, and timestamps are durably stored in S3 in the user-metadata of the object associated with the file. Once objects are transferred to S3, they can be managed as native S3 objects, and bucket policies such as versioning, lifecycle management, and cross-region replication apply directly to objects stored in your bucket.

Volume Gateway

• The volume interface presents your applications with disk volumes using the iSCSI block protocol.
• Data written to these volumes can be asynchronously backed up as point-in-line snapshots of your volumes, and stored in the clouod as Amazon EBS snapshots.
• Snapshots are incremental backups that capture only changed blocks. All snapshot storage is also compressed to minimize your storage changes.

Volume Gateway - Stored Volumes

• Stored volumes let you store your primary data locally, while asynchronously backing up the data to AWS.
• Stored volumes provide your on-premises applications with low-latency access to their entire datasets, while providing durable, off-site backups.
• You can create stored volumes and mount them as iSCSI devices from you on-premises application servers.
• Data written to you stored volumes is stored on your on-premises storage hardware.
• This data is asynchronously backed up to Amazon Simple Storage Service (Amazon S3) in the form of Amazon Elastic Block Store (Amazon EBS) snapshots.
• 1 GB - 16 TB in size for Stored Volumes.

Volume Gateway - Cahed Volumes

• Cached volumes let you use Amazon Simple Storage Servie (Amazon S3) as your primary data storage while retaining frequently accessed data locally in your storage gateway.
• Cahed volumes minimize the need to scale your on-premises storage infrastructure, while still providing your applications with low-latency access to their frequently access data.
• You can create storage volumes up to 32 TiB in size and attach to them as iSCSI devices from your on-premises application servers.
• Your gateway stores data that you write to these volumes in Amazon S3 and retains recently read data in your on-premises storage gateway's cache and upload buffer storage. 1 GB - 32 TB in size for Cached Volumes.

Volume Gateway - Tape Gateway

• Tape Gateway offers a durable, cost-effective solution to archive your data in the AWS cloud.
• The VTL interface it provides lets you leverage your existing tape-based backup application infrastructure to storedata on virtual tape cartridges that you create on your tape gateway.
• Each tape gateway is preconfigured with a media changer and tape drives, which are available for your existing client backup applications as iSCSI devices.
• You add tape cartridges as you need to archive your data.
• Supported by NetBackup, Backup Exec, Veeam etc.

Types of Snowballs

• Snowball
• Snowball Edge
• Snowmobile

Snowball

• Snowball is a petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data into and out of AWS.
• Snowball addresses common challenges with large-scale data transfers:
  • Network costs
  • Long transfer times, and security concerns.
• Transferring data with Snowball is simple, fast, secure, and can be as little as one-fifth the cost of high-speed internet.
• 80TB snowball in all regions.
• Snowball uses multiple layers of security designed to protect your data including tamper-resistant enclosures, 256-bit encryption, and an industry-standard Trusted Platform Module (TPM) designed to ensure both security and full chain-of-custody of you data.
• Once the data transfer job has been processed and verified, AWS performs a software erasure of the Snowball appliance.

Snowball Edge

• 100 TB data transfer deviec with on-board storage and compute capabilities.

Snowmobile

• Exabyte-scale data transfer service.
• Can transfer up to 100PB per Snowmobile.

S3 Transfer Acceleration

• Utilizes the CloudFront Edge Network to accelerate your uploads to S3.
• Instead of uploading directly to you S3 bucket, you can use a distinct URL to upload directly to an edge location which will then transfer that file to S3.