This is the note for AWS course 1 on edX. List selected AWS services in alphabetic order. Course link is here.
Building on AWS --
- Create VPC which simulates a local network that contains all your servers and databases.
- Create IAM policy, user/role which has permission to specific AWS services, but not all -- this adds security overall.
- Create S3 bucket (login as specific IAM user) which will be used to store assets.
- Create RDS database instance (login as specific IAM user) as database server.
- Create Cloud9 environment (login as specific IAM user) which is an online IDE that you can build & save your project.
- Create EC2 instance (login as specific IAM user) and deploy the application via user data. This EC2 instance should have corresponding IAM role (to allow EC2 instance to call AWS service) and security group (to allow web traffic). This instance and its security group are within the VPC, and instance should be in a public subnet.
- Create and configure the Application Load Balancer to minimize downtime.
- Create Cognito user pool via Amazon Cognito to manage user authentication.
Security & encryption
- Deploy SSL/TLS certificates to AWS infrastructure using AWS Certificate Manager (ACM), see official documentation here, also check course material wk4.
- Secure the connection between user and load balancer by adding listener that uses encrypted connections. See course material wk4 and documentation here.
- Check course material wk4 for data encryption.
Debugging & profiling
- Use AWS X-Ray to analyze traces of the application for debugging and finding opportunities to optimize. Check course material wk5.
Go serverless with Lambda
- Create Amazon Lambda to reduce server side code (or go serverless) and run async operations.
Distribute events to on-premises application
- Use Amazon Simple Notification Service (SNS) to create a topic that pass the event to multiple applications subscribing to the topic. Then use Amazon Simple Queue Service (SQS) to handle the message queue that will be consumed by distrubted applications. Check course material wk6.
Install AWS CLI follow official guide (https://docs.aws.amazon.com/cli/latest/userguide/awscli-install-windows.html), or verify AWS CLI has been installed by typing the following in your terminal
aws --version
It should output something like below
aws-cli/1.11.84 Python/3.6.2 Windows/7 botocore/1.5.47
A scalable service for creating and maintaining a user directory and user authentication. You can create a Cognito user pool to manage user authentication. Cognito user pool is linked to your application via application URL.
Cloud-based integrated development environment (IDE) that lets you write, run, and debug your code with just a browser.
When creating a new Cloud9 envrionment, you should select the VPC (Configure settings -> Network settings (advanced) -> Network) where other related servers reside so that the environment can access them.
EC2 is a virtual server that you can treat it as a virtual computer that can run programs, install packages, build and deploy apps, etc. Creating an EC2 instance is like getting a virtual computer for later use. When creating EC2, you can associate it with specific VPC (and subnet) and IAM.
Prerequisites: create at least 2 EC2 instances and deploy the application on them. The two EC2 instances should be inside same VPC but different public subnet (different availability zones).
In the EC2 dashboard, find LOAD BALANCING, then Load Balancers -> Create Load Balancer -> Application Load Balancer -> Create. Enter a name and keep default settings for Scheme and Listeners. In Availability Zones, for VPC, select the VPC of your projects; for availability zones, select all AZs on the list, and select the public subnet for each AZ. Skip configures until Assign a security group, choose the same security group as your web server EC2 instance. For Configure Routing, for target group, click New target group, type a name and keep default settings and click next to Register Targets. For instances, select the 2 (or even more) EC2 instances you created for the deployment, then Add to registered, review and create.
After the load balancer becomes active, the web traffic will be routed via load balancer to the servers. If you paste the DNS name of the load balancer to browser, you should see your application. Since there're two servers registered, if one server is down, the other working server will take the traffic so as to minimize downtime.
First SSH into the existing EC2 instance, then To view the log file, type the following in the instance terminal
cat /var/log/cloud-init-output.log
To view instance metadata,
curl http://169.254.169.254/latest/meta-data/
To get public IP address,
curl http://169.254.169.254/latest/meta-data/public-ipv4
To get instance MAC address,
curl http://169.254.169.254/latest/meta-data/mac
To get the VPC ID in which the instance resides. Make sure to replace Your-MAC in the command below with the MAC address of your instance,
curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/Your-MAC/vpc-id
To get the subnet-id in which the instance resides. Make sure to replace Your-MAC in the command below with the MAC address of your instance,
curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/Your-MAC/subnet-id
To get instance identity document,
curl http://169.254.169.254/latest/dynamic/instance-identity/document
To get the instance user data,
curl http://169.254.169.254/latest/user-data
IAM is the place you can create users and set permissions for each user. Inside IAM you can create users, groups and roles.
First create an IAM policy. Check the AWS IAM policy template here. Or add exisiting policy by typing keywords in the search box.
Then create an IAM user, attach the policy and generate access key. Important: Download the .csv file with the access keys after creating the user. Also, make sure to click the Send email link to send email instructions to your email address.
To use IAM user, log out AWS account and sign in again using the link provided in the email instruction, and the username and password of the user you created. Now you'll have permission to do whatever this user is permitted to do by the policy (such as create EC2 instance, etc). Root user will have full permission to everything the IAM users created.
IAM role is different from IAM user. It's not a "controller" or "creator" of the project, but rather part of the project. For example, an EC2 instance can take a specific IAM role, which allows EC2 instance to call AW2 services on your behalf. Details see the following.
In AWS console, enter IAM service, click Roles and then Create role. For Select type of trusted entity, select EC2. For Select your use case, select the first option: Allows EC2 instances to call AWS services on your behalf. For Permissions, select what you need. Review, assign role name and create role.
Use the role when you create new EC2 instance.
On the instance terminal, type the below command
aws configure
You'll be prompted to enter Access Key Id and Secret Access Key. Open the credentials.csv file that you downloaded when creating IAM user to find these information. For Region, type corresponding region (us-west-2). For Default output format, press ENTER. To query info about current EC2 instance, type the following command
aws ec2 describe-instances
Check course material wk5. Lambda runs your customized code when it is triggered. In creating an Lambda function, you need to
- create a Network Access Instance (NAT) in your VPC (so that Lambda can access internet from your private subnet - yes, Lambda should be placed in a private subnet when you create the function),
- create an IAM role for Lambda so it can access relevant AWS services used in the code (if applicable),
- create a security group for Lambda to access AWS database used in the code (if applicable),
- write the code to be run directly, or by uploading the code contained in a .zip file, by the following command in AWS CLI
aws lambda update-function-code --function-name <your function name> --zip-file fileb://<your zip file including extension>
- add environment variables,
- configure a trigger to the Lambda function. For example, for a S3 instance, go to Properties -> Advanced settings -> Events -> Add notification -> Events, select relevant option -> Send to, select Lambda Function -> Lambda, select the function you created -> Save
- test the app
In RDS users can create relational database instances such as MySQL. Note that you should create the database instance within the same VPC of other servers in the project, but in a private subnet so as to avoid attack from outside VPC. Make a note on database name, Master username, password, database name and database endpoint, as they will be used in order to connect to the database instance. You may also need to update the security group under Security and network.
An object storage. To download object from S3 bucket, type the following in an EC2 instance terminal (or Cloud9 environment terminal), be sure to replace with the target object link,
wget <link to the object>
And unzip the file if necessary,
unzip <filename including extension>
To add files to compressed file, the following command will create a zip file under root directory. The zip file will contain folder1, folder2 and file1, file2, where file1, file2 is under the current work directory.
zip -r ~/<zipfile name including extension> folder1/ folder2/ file1 file2
To upload the zipped file to S3 (current IAM user must have written permision to S3).
aws s3 cp ~/<zipfile name including extension> s3://<YOUR_BUCKET_NAME>/
VPC is a network that you can put your virtual servers and databases in. All instances inside are connected within VPC, and there is one internet gateway that connect the VPC to internet. Inside VPC, two groups of subnets can be configed: one public subnet (for example web servers), which is accessible via internet from outside the VPC, and one private subnet (for example databases), which is not accessible from outside the VPC. The purpose of VPC is to add additional security layer.
VPC can be easily created using CloudFormation, an AWS service that can take 'template' to provision AWS resources.
Once have VPC set up, you can create EC2 instance that belongs to a specific VPC and its subnet.
Use attached template file from edX AWS: OTP-AWSD1 course, which contains 2 public subnets and 2 private subnets with corresponding names.
In AWS console, enter CloudFormation and then Create Stack with the template (here. Skip the Options page and keep all the defaults to Create.
Once VPC is created, it can be used when creating new EC2 instances.
From official documentation --
You can use a network address translation (NAT) instance in a public subnet in your VPC to enable instances in the private subnet to initiate outbound IPv4 traffic to the Internet or other AWS services, but prevent the instances from receiving inbound traffic initiated by someone on the Internet
For example, add a NAT instance in VPC for AWS Lambda. See AWS VPC template with NAT for CloudFormation from course material.
Image and video recognition service. It supports functionality such scene detection, face detection, even celebrity recognition.
Lifelike text-to-speech service.