We have remote developers who occassionally need access to AWS servers QA and Staging databases (RDS mysql instances). The AWS servers (EC2, fargate) are in a private VPC. The RDS databases are in different VPCs, they have the "publicly accessible" attribute set, which means they get a pubilc DNS, but only a handful or IPs are whitelisted for that access; developers should get access over a VPN.
This is summarized as:
laptop --ClientVPN--> VPC _A_ --VPC Peer--> RDS in VPC _B_
I choose the Cliet VPN Endpoint so that AWS would manage the remote side of the tunnel. I choose Viscosity (on a Mac) as our VPN client because it's easy to use and support split-dns and split-routing. It's affordable, but not free. Split DNS is important so that Amazon hostnames can be resolved to their internal IP addresses. Split routing is important so that only the AWS destined traffic goes over the VPC tunnel and other internet traffic can go direct to internet.
You can read the AWS getting started docs for Client VPN endpoint, but there's a lot of details. This checklist can be helpful to troubleshoot or if you have a slightly different scenario from what AWS's tutorial covers, such as the arrangement I describe above.
NOTE: This is a trouble shooting / set up checklist; not full documentation. Start with AWS's getting started to walk you through the steps of creating the basic endpoint.
NOTE 2: There are a few things I wasn't certain are required. They are labelled with a ? and a question.
After creating the Endpoint, RDS instances, VPCs, etc, gather the relevant details for reference later.
- Client VPN Endpoint (cVPN).
Id _________________________ / Name _________________________
client cidr block _________________________
- Associated VPC
Id _________________________ / Name _________________________
CIDR block _________________________
- Associated Subnet
Id _________________________ / Name _________________________
CIDR block _________________________
- Associated Security Group
Id _________________________ / Name _________________________
- Route table for the subnet that is assocaited to the VPN
Id _________________________ / Name _________________________
- For each Peering connection:
Id _________________________
Peered VPC: Id _________________________ / Name _________________________ / CIDR _________________________
-
Associations: Atleast 1 association to a subnet and a security group in that subnet
-
Subnet - subnet's CIDR block must NOT overlap cVPN's client cidr block
-
Security Groups
- Inbound, All traffic from same security group (self)
- ? Inbound, All traffic from the subnet's CIDR block (is this required?)
- ? Inbound, All traffic from the cVPN's CIDR block (is this required?)
-
Authorizations (all the cidr blocks that user will want to access via the VPN)
- ? VPNs client cidr block (is this required?)
- For each VPC Peering connection: CIDR block for acccepter side VPC
- 0.0.0.0/0 for All (to authorize access to internet gateway) (? Not sure if this is required?)
-
For each VPC Peering connection
- The accepter VPC must have a return route for cVPN's subnet
- If you want DNS to resolve to private IP addresses (necessary in my scenario described above), be sure to enable "Allow requester VPC (...) to resolve DNS of accepter VPC (...) hosts to private IP" for the Peering Connection.
-
cVPN Endpoint Route Table
- The cVPN's associated VPC CIDR
- The peered VPC's CIDR
-
Route Table associated with cVPN's associated subnet
- The cVPN's VPC CIDR to "local"
- 0.0.0.0/0 for All to Internet Gateway (igw-*)
- For each VPC Peering connection: CIDR block for acccepter side VPC
Create a config by importing the client config (ovpn file) that you download from AWS. Then edit the config (Viscosity -> Preferences) as follows:
-
In advanced options, add:
route-nopull -
Under "Networking -> Routing", add
- Route for VPN's VPC cidr block, gateway = vpn_gateway, and
- Zero or more routes like this: Route for peered VPC's cidr block, gateway = vpn_gateway
- Under "Networking -> DNS Settings, set
Mode: Automatic
Server: ___ Use the ".2" address in the VPN's client cidr block ___
Domains: amazonaws.com
Things can get tricky for complex set ups, so it's best to test one step at a time. The specific steps might vary for you, but here are some of the stpes (connections) that I tested for the scenario described above.
- Access to VPC A (from an EC2 instance-- if your VPC does not have an instance create one a "temp" one w/ public IP and a security group that allows acces from within the VPC)
ssh -i KEY_FILE ec2-user@___PRIVATE_IP_ADDRESS_A__
- Access to VPC B (some EC2 instance, or create a "temp" instance if needed) using private IP from host in VPC A
(from instance in previous step)
ssh ec2-user@___PRIVATE_IP_ADDRESS_B__
# It's OK if auth fails b/c you don't have SSH key, this still proves net access is OK
- Access to VPC B (Like #2, but this tieme directly from your laptop) using private IP
ssh -i KEY_FILE ec2-user@___PRIVATE_IP_ADDRESS_B__
# It's OK if auth fails b/c you don't have SSH key, this still proves net access is OK
- telnet on mysql port to rds (in VPC B), using private IP
telnet __PRIVATE_IP_OF_RDS_INSTANCE__ 3306
# Port 3306 is for mysql
# You don't need to complete login, just see that RDS responds, for example, Mysql asks for password.
# A failure will be a TIMEOUT error
# Alternatively, you can use `mysql` command line tool to test access
- telnet on mysql port to rds (in VPC B), using DNS for public hostname resolution (testing DNS for amazonaws.com domain).
Note:
nslookupmight bypass the split-DNS set up b/c it uses raw sockets. But applications (liketelnetandmysql) should work, so we test with that. You can also usedig HOSTNAME @DNS_SERVER_IPto help test.
telnet __sample.afb8917naaccc.us-east-1.rds.amazonaws.com__ 3306
# See notes on `telnet` above.