Skip to content

Instantly share code, notes, and snippets.

@brijesh-deb

brijesh-deb/AWS_Note.md Secret

Last active February 6, 2018 06:51
Show Gist options
  • Save brijesh-deb/aeff81c24b94490aa00d7a48a2c30c99 to your computer and use it in GitHub Desktop.
Save brijesh-deb/aeff81c24b94490aa00d7a48a2c30c99 to your computer and use it in GitHub Desktop.
Download ZIP
#AWS #Note
Raw
AWS_Note.md

AWS Quick Notes

  • ARN : Amazon Resource Name
  • Ways to access AWS: Web console, CLI, SDK
  • For access AWS thru CLI or SDK, we need Access Key ID and Secret Access Key of the account.
  • SNS topics are region specific and can only be notified by alarms in same region.
  • IAM
    • IAM User is an entity that is created in AWS to represent the person or service that uses it to interact with AWS.
    • A user in AWS consists of name and credentails.
    • IAM Role is similar to user, an AWS identity with permission policies that determines what can and cannot do in AWS. However, instead of being uniquely associate with one person, a role can be assigned to anyone who needs it.
    • A role does not have stanadard credential(password or access keys); instead if a user or service assumes a role, temporary security credentials are created and provided.
    • Role can be used to delegate access to users, applications or services that dont normally have access to AWS resources. For example, if your application deployed on EC2 instance has to access S3, instead of providing credentials(access key and secret), create a role that has access to S3 and assign that role to the EC2 instance. This way credentials is not exposed, also only specific EC2 instance will have access to S3.
    • IAM Policy is a collection of permissions to access different services in a particular way
    • Policy has 3 parts ( can be found in JSON structure]
      • Action: Operation that user can perform
      • Effect: Allow or Deny; Deny by default
      • Resource: Specific resources(having ARN) uses can perform action on
    • Its better to apply Policies to Group and add Users to the Group; rather than applying Policies to individual Users.
  • Multi Factor Authentication (MFA) ensure that an user i)"Knows" a Password ii)"Have" a device like smartphone; correct password has to entered from a specific device
  • Regions and Availability zones
    • Not every region and AZ supports every AWS resource
    • Prices for AWS usage vary by region
    • Resources are not replicated across regions automatically.
  • VPC and Subnets
    • VPC is a virtual network dedicated to your AWS account, logically isolated from other virtual networks in AWS cloud.
    • VPCs use routing tables to configure routing destinations for traffic coming out of VPC. Each VPC has 1 routing table
    • Subnet is a segment of a VPC's IP address that you can launch instances into. Enables you to group instances based on security and operational need.
    • Subnet can exist in a single Availability Zone(AZ).
    • A VPC can be connected to internet through Internet Gateway. A subnet is deemed a Public Subnet if it has a Route Touble that directs traffic to the Internet Gateway.
    • Only 1 Internet Gateway can be associated with a VPC.Its a logical connection and not a physical device.
  • NAT Instance and NAT Gateway
    • Instances in a private Subnet can access Internet using NAT Instance or NAT Gateway
    • NAT Instance is an EC2 instance configured to forward traffic to internet (since it is in a Public Subnet).
    • AWS introduced a NAT Gateway Service that can be used in place of NAT instance. Benefit is that its a fully manged service, has fail-over.
  • Security Group
    • Defines allowed incoming/outgoing IP addresses and ports. Kind of mini firewall.
  • EC2 instances can be Compute Optimized or Memory Optimized
  • Amazon Route 53
    • Is a highly available and scalable cloud Domain Name System(DNS) web service. Translates domain names (such as www.example.com) into numeric IP addresses that computers use to connect to each other.
    • Lets you register domain name for your website, routes traffic to resources in your domain.
    • Enables you to organize DNS records using hosted zones. Hosted zone is a collection of resouce record sets for a specified domain.
    • Resource record sets tell DNS how you want traffic to be routed for that domain.
  • Auto Scaling Group
    • Maintains a fleet of EC2 instances that can scale up/down to handle changing load.
    • Can span multiple AZs to protect against potential failure of a single AZ.
    • ELB distributes traffic evenly among EC2 instances in the group.
    • When group launches or terminates instaces based on load, load balancer automatically adjusts.
    • Auto scaling is region specific and only between AZs in that region. It's not even a good idea to auto-scale across regions. Auto-scale is more geared for a specific tier of your application. For example, if a region was to go down, you would not want some of your web servers to use services across a slow link to another region (potentially) across the country. Instead you would want route 53 to route the traffic to an autonomous stack running it's own auto-scaled layers in a separate region.
    • Auto scaling components: i) Groups ii) Launch configurations iii) Scaling plans
    • You can create as many Auto Scaling group as needed for your architecture, like an Auto Scaling group for each tier.
    • Auto scaling groups cannt span multiple regions.
    • Rebalancing: After certain actions occur, Auto scaling group can become unbalanced between AZs. Auto Scaling compensates by rebalancing the AZs.
    • When rebalacing, Auto Scaling launch new instances before terminating the old ones, so that rebalancing does not compromise the performance or availability of your application.So the system can temporarily exceed specified max capacity of group by 10 percent( or by 1 instance margin which ever is greater) during rebalancing. This extension lasts only as long as needed to rebalance the group typically a few minutes.
    • Ec2 instances in an auto scaling group have a path or lifecycle that differs from that of other EC2 instances. You can add a lifecycle hook to your auto scaling group so that they can perform custom actions when instances launch or terminate.
  • Storage options provided by AWS
    • AWS provides following storage
      • Instance Store (Ephemeral Storage):
      • Elastic Block Store (EBS): Similar to computer drive but in virtualized environment.
      • Simple Storage Service (S3): object storage suitable for storing user files and backups.
      • Elastic File System (EFS): scalable storage that can be accessed from multiple EC2 instance.
    • EC2 instances can be launched using either EBS or Instance Stire as root volumnes and additional volumes. AMI includes the choice of root volume.
    • Trade off between performance, cost and access.
    • S3 is cheapest, write-once, read-multiple times
    • EFS is much cheaper than EBS, but EBS can only be accessed by 1 EC2 instance at a time.
    • EBS can give better performance than EFS, but EFS gains a point on EBS as it can be used a shared network storage like NAS.
    • S3 is slower than EFS as it uses eventual consistency.
  • Instance Storage
    • Accesses storage from disks that are physically attached to host computer.
    • Data persists only during the life of associated instance; if instance is stopped or terminated data is lost.
    • Data is not lost when instance is rebooted.
    • Instance store volume can be attache as additional volume only when instance is being launched and not when instance is up and running.
    • Use Instance store volumes for temporary storage of data that changes frequently such as buffers, caches or scratch data or data that is replicated across a fleet of instances.
    • If data must persist beyond lifetime of EC2 instance, use EBS volumes instead.
  • Elastic Block Store (EBS)
    • Storage for drives of virtual machine i.e EC2
    • Not a standalone storage service like S3, so it can only be use in combination with EC2.
    • Only allows the volume to attach to another EC2 instance or keep it in a standby mode.
    • Once configured volume in EBS, it cannt be easily scaled. If you need more storage space, you will need to buy and configure new volume of a bigger size.
    • Can be attached as additional volume when instance is launched and even when instance is running.
    • Each volume can be attached to one and only one instance.
    • Boot time is faster than Instance Storage.
    • Data can be backed up by creating snapshots, new EBS volume can be created from the snapshot later.
    • EBS volume types (based on IOPS)
      • General Purpose (SSD) volumes: Upto to 10,000 IOPS; good fit for lot of read/write opeartions like Postgres, Oracle.
      • Provisioned IOPS (SSD) volumes: upto 20,000 IOPS
      • Magnetic volumes: IOPS within 100 to hundreds of IOPS. Used for applications that dont require lot of read/write operations.
    • IOPS stands for input/output operations per second.
  • Simple Storage Service (S3)
    • Stores object and allow to easily configure who can see them
    • Objects are fundamental storage structure. An Object is basically a file and metadata.
    • Can store objects in which ever region selected.
    • Max object size that can be stored is 5 TB, with 5 GB for single put operation.
    • Objects are stored in Bucket
    • Buckets create a URL namespace for objects stored in it. For instance a bucket create in Oregan region with name "xxxx" will have an URL s3-us-west-2.amazonaws.com/xxxx as the root for objects. That's why bucket name should be unique across all of S3 and not just ur account.
    • Objects in S3 are uniquely identified by key which is Path and Filename; excluding bucket URL. So an object xyx.png in folder imanges will have object key: images/xyz.png.
    • Cross-region replication replicates any file in S3 to another region. This can help reduce latency.
    • Highly scalable with "eleven nines" data durability (99.999999999%)
  • Elastic File System(EFS)
    • EBS is good for setting up drive for virtual machines while S3 is good for storage. EFS fulfills middle ground.
    • Suits applications with high workloads that needs scalable storage and relative fast output.
    • EFS is automatically scalable.
  • S3 vs EBS vs EFS
S3 EBS EFS
Can be publicly accessible Accessible only via the given EC2 Machine Accessible via several EC2 machines and AWS services
Web interface File System interface Web and file system interface
Object Storage Block Storage Object storage
Scalable Hardly scalable Scalable
Slower than EBS and EFS Faster than S3 and EFS Faster than S3, slower than EBS
  • Glacier
    • Suitable for data archiving and backup.
    • Vault is a container for archives, an archive is any object that is stored in vault.
    • Vault can be created/deleted from management console, but archieves can not be uploaded/downloaded/deleted from management console. Use CLI or write code.
    • Archieve operations(upload/download) takes 3 to 5 hours to complete.
  • For large binary files(audio,video and image) consider storing in S3 and storing metadata for the files in database.
  • AWS RDS
    • DB instance is the basic building block of RDS, its an isolated database environment in cloud. A DB instance can contain multiple database.
    • When a DB instance is created, under the hood AWS creates an EC2 instance and installs OS(linux) and DB engine like PostGres. That's why size of the EC2 instance that powers the db is very important.Underpower EC2 instance wont be able to process queries.
    • Uses EBS volumes internally for storing data.
    • In multi-AZ deployment, RDS automatically provisions and maintains a synchronous standby replica in a different AZ.
    • Provides 2 different methods for backing up and restoring DB instances: i) automated backup ii) user-initiated backup using DB snapshots.
  • CloudFront Distributions
    • Helps create a content delivery network (CDN) that makes website content available from data centers around the world, called edge locations.
    • Store content on an origin serer, such as S3 bucket or HTTP server running on an EC2 instance. Create a CloudFront distribution and then associate the distribution with the origin server. Then content can be accessed using CloudFront URL.
    • When user access an object that's part of CloudFront distribution, CloudFront checks whether the object is already in a cache near the user. If it is, content is served from the cache; other CloudFront copies the content from the origin server to the cache and servers it to user.
  • Distributing load to multiple nodes
    • Push model: Use a load balancing solution like ELB to route incoming requests across multiple EC2 instances.
    • Pull model: Asynchronous event driven workloads do not require a load balancing solution. Instead tasks to be performed are stored as message in a queue (like AWS SQS). Multiple nodes can then pull and consume those messages, processing them in distributed fashion.
  • Session Management
    • When user uses a web page, a user session is created by the server and the session ID is managed internally during web session of the user.
    • Possible solutions on AWS
      • Sticky Session
      • NoSQL Dynamo DB
    • Sticky Session: ELB provides feature to enable stikcy session (or session affinity), needs no application code changes. Ensures that ELB sends all requests from a single user to the same server where session is created. This is against principles of horizontal scalability.
    • Dynamo DB: Session details can be stored in Dynamo DB against session ID. Data is automatically replicated across 3 AZ in a Region thus providing high availability and durability.
  • Immutable Infrastructure
    • An infrastructure where server that once deployed is never modifie, merely replaced with a new updated instance.
    • Advantages
      • Simplified operation in an auto-scaling environment.
      • Continous deployment, fewer failures
      • Less down time
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment