Assume you've been asked to create a VM on AWS to run some critical operations for your business; it needs to access the internet, but only can be accessed by the maintainers (e.g. people/services who would want to install/upgrade the software). How would you do it?
This is a series of 2 posts. In the first post, we went over what happens when we create an EC2 instance (VM) in AWS, where we explained how the instance gets attached to the default VPC and the traffic gets routed.
In this second post, we will go over creating a full secure network where the newly created VM will operate.
Why can't we just use the default vpc? The default VPC is available to get users started quickly without thinking about the underlying network. So it works great if you are doing some simple deployments, or If you are testing or experimenting in AWS.
The default VPC has multiple components associated with it, such as subnets, NACLs, Security Group and Route table. Some of these component are open publicly, which does not form the most solid security practice.
Moreover, using the default VPC will limit the user to using the 172.31.0.0/116 range. By having a custom VPC the customer will be able to tailor the network exactly and avoid overlapping IP addresses, including on-premise networks
It is recommended to configure a custom VPC, where the user has full control over the network model and its components. This makes it easier to scale the application and architecture within private subnets.
In this section we will start by creating the necessary resources progressively using cloud formation. At every step of this section, I suggest you upload the template to CloudFromation and have a look at the resources that get created with the stack. Use the following command at your convenience (or upload the template manually from the console):
To create the stack - aws cloudformation create-stack --stack-name private-network --template-body file://infra/template-custom-vpc.yml
To update the stack - aws cloudformation update-stack --stack-name private-network --template-body file://infra/template-custom-vpc.yml
The architecture would look like the following diagram, where we only allow outbound access from our VPC - we will cover how to enable inbound access for the VM maintainers in the next section and update the diagram accordingly, 
Below are the details on how to create each component of this diagram.
Use the following template to create the custom VPC:
https://gist.github.com/007e9d065171883c74224ed55bce6f13
In this previous snippet, we have given a name for the VPC, a CidrBlock that defines the range of IPS that could be utilised (in this case from 192.168.0.0 to 192.168.255.255). When setting the EnableDnsSupport and EnableDnsHostnames flags to true, the instances with a public IP address get corresponding public DNS hostnames, and the Amazon Route 53 Resolver server can resolve Amazon-provided private DNS hostname. If you're curious to know more check the detailed AWS documentation: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html.
In the template file, under "Resources", add the following:
https://gist.github.com/d7af6908f04356d1c598034a6693ed25
This would create the internet gateway, but it won't be attached to a VPC: 
Add the following resource to your template to attach the internet gateway to the created VPC:
https://gist.github.com/dcc7bd6f6bcd8a3840d809adb728a10e
Afterwards, we will create 2 subnets, one private and one public - the private one will contain our EC2 instance and the other can connect to the internet through the internet gateway.
https://gist.github.com/8a03666d94dd21656e00c6bcd1cb7ebb
When creating a subnet we do not deicde whether it's private or public through a configuration on creation, so for this step these are the subnets that are intended to be private and public; we will configure them for that in future steps: 
Notice how we split the Cidr blocks of the VPC into 2 Cidr blocks (one for each subnet). The Availability zone property uses 2 CloudFormation intrinsic functions:
- GetAZs, which returns a list of all of the Availability zones for a specific region; specifying an empty string is equivalent to specifying the region in which the CloudFormation stack is created
- Select, which returns a single object from a list of objects by index
In this case, we are getting the first availability zone in the us-east-1 region, in which we're creating the stack.
The MapPublicIpOnLaunch property in the public subnet indicates whether instances launched in this subnet receive a public IPv4 address. This might be confusing as we discussed creating the EC2 instance in the private network, so why are we adding this configuration? Briefly, we will create another EC2 instance in the public subnet, which would be a utility to SSH into the private subnet instance later on. We will get back to more details about this instance in the coming steps.
We will now create a route table and associate it with the public subnet, in addition to a default route
https://gist.github.com/83bd056aceac2a3db234d63b7f8e06ba
Looking at the created route table, we will see it associated with the public subnet, and has the following routes - the record with 0.0.0.0/0 as a destination will target the internet gateway created: 
The NAT(Network Address Translation) is the method of giving a private resource access to the internet. It always uses Elastic IPs (Static public IPs).
https://gist.github.com/6636b8381e57b29bb3dd2ea98785a307
The NAT gateway we created is associated with the public subnet`
As we have created the NAT gateway, we will now create the route table associated with the private network, with a route targeting the NAT gateway. This should look like this:
https://gist.github.com/97212d479ab18a7ee8f2d25cb970a0fa
In the previous section, we have setup the VPC and necessary components that would allow the interaction of our private VM with the internet. Let us create the private EC2 instance and its associated security group, then try to connect to it using SSH:
https://gist.github.com/b5c343f1321ef3e7a9a1ae5a01600b37
This stack would fail because of the KeyName property, which was not passed anywhere. Follow the following steps to fix this:
- Create a key pair to be used to ssh to the EC2 instance: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#prepare-key-pair
- Add the following before the "Resources" section of your cloud formation tempate
Parameters: KeyName: Description: EC2 KeyPair Type: AWS::EC2::KeyPair::KeyName - When creating the CloudFormation stack pass the key - you can use the following in the command line:
aws cloudformation create-stack --stack-name private-network --template-body file://infra/template-custom-vpc.yml --parameters ParameterKey=KeyName,ParameterValue=<Name of the key your created in the first step>. Alternatively, you can manually pass that param if you're using the console to upload the template and create the stack.
Perfect, by now you should have an EC2 instance running in the private subnet. Right click it in the console and click "Connect". In the "EC2 Instance Connect" tab, click the "connect" button --> Notice you got an error, even though that EC2 instance has a security group attached that allows SSH connections. But as it is in a private network, we wouldn't be able to SSH to it from the internet.
In this section we will build the component(s) needed to allow the maintainers to SSH into the private EC2 instance, which we will create in the last section.
In the subnets section we mentioned the creation of an EC2 instance in the public subnet. That would be the bastion host. It is a server whose purpose is to provide access to a private network from an external network (e.g. the internet).
That's the only addition to our architectural diagram to allow ssh traffic from maintainers: 
https://gist.github.com/dad2d1de6daaf96bb12678c780d49f06
So in order to connect to the internal (private) instance, the user (maintainer) would want to ssh to the bastion host, and from it ssh to the private instance. Have a look at the following article for more details about SSH Agent Forwarding, but if you like to do it quickly there's a summarized version below: https://aws.amazon.com/blogs/security/securely-connect-to-linux-instances-running-in-a-private-amazon-vpc/.
Below is a summary of what we need to do:
- Verify the SSH agent is running on your OS:
eval ssh-agent(on MacOS) - If you got Agent pid with a process number, it means it's running - Add the private part of the key to the agent: assuming my key is called ec2-instance.pem, the command on MacOS would be:
ssh-add -K ./ec2-instance.pem. You should get an output saying Identity added. - Go the AWS console, right-click the newly created instance (Bastion host), and click "Connect"
- In the "SSH client" tab, follow the instructions to ssh from your terminal
- In the last command, we would need to updated it to configure SSH forwarding. So instead of
ssh -i "ec2-instance.pem" ec2-user@ec2-.....compute-1.amazonaws.com, run the following command:ssh -i -A "ec2-instance.pem" ec2-user@ec2-.....compute-1.amazonaws.com - If no errors appeared, we will now be connected to the Bastion host instance
- Now let's try to connect from the Bastion host through to the EC2 instance in the private network (Get the connection information the same as we did for the Bastion host) - In the last command in the SSH Client tab, delete the -i and the name of the key. So the command would look like this
ssh ec2-user@ec2-.....compute-1.amazonaws.com - Now you can see we are now on the private instance. Try running
ping 1.1.1.1and notice how the instance has access to the internet.
That's it for the day. I hope this was helpful and would love to hear any feedback, comments or questions.