Skip to content

Instantly share code, notes, and snippets.

@WintersMichael

WintersMichael/create_kubernetes_cluster_with_kops_and_terraform.md

Last active June 12, 2018 19:48
Show Gist options
  • Save WintersMichael/0ab96ae9ba4fc2afe48e5a7294a1c70d to your computer and use it in GitHub Desktop.
Save WintersMichael/0ab96ae9ba4fc2afe48e5a7294a1c70d to your computer and use it in GitHub Desktop.
Download ZIP
Create a Kubernetes cluster at AWS with kops and terraform
Raw
create_kubernetes_cluster_with_kops_and_terraform.md

Create a Kubernetes cluster at AWS with kops and terraform

This is how I do it. Considerations going into this approach are:

  • It puts all of your resources under terraform control, making it easy to reuse terraform code from other projects and/or use terraform for further integrations.
  • It permits a fair amount of flexibility in workstation configuration (no need to hardcode your workstation to match mine).
  • It provides a good setup for locally switching between multiple projects which might not all be running the same versions.

The examples here will assume that you are going to create a dev cluster for a fictional product named "Bigmoney". It is suggested to not use your product name in your cluster names, since product names are at the whim of marketing and other business tides. Instead, we will use a project codename of "ampersand" to signify "my team's clusters".

Workstation setup

  1. Install and configure direnv.
  2. Download the latest version of terraform and an appropriate version of kops and kubectl for the kubernetes version you wish to run, and place them somewhere outside your PATH. Eg, ~/bin/kops/1.8.1/kops
  3. Make sure you have AWS CLI credentials to your target account. This is outside the scope of this document, but suggested setup is to have named profiles in ~/.aws/credentials, eg "ampersand-nonprod" matching an AWS account of the same name, for ease of switching between accounts.

Planning

S3

We will be creating an s3 bucket in the target AWS account to contain the terraform remote state and the kops configuration. This example will use bucket "ampersand-dev-remote-state".

Route53

We will be creating a route53 domain in the target AWS account, which will essentially be "owned" by the kubernetes cluster. Entries in this domain will be created by k8s based on what you deploy inside the cluster, though you can also create additional entries manually. If this is your prod cluster, it is suggested not to use the k8s-generated DNS entries as your public endpoints - instead, use CNAMEs from your public "written in stone" address to the cluster-generated endpoint, as this provides future flexibility for blue/green and other internal needs.

If your cluster domain is going to be a subdomain (it almost always is), you will need to delegate control from the parent domain, so make sure you have access to that. This example will use a cluster domain of "ampersand-dev.bigmoney.mycompany.com", so in this case you would need to make sure you have access to "bigmoney.mycompany.com" or know someone who does.

Availability Zones

Most AWS accounts do not have access to all AZs. You can find out which AZs your account has access to via the aws CLI: aws ec2 describe-availability-zones. (You may need to export AWS_PROFILE=ampersand-nonprod if this is your first time accessing the AWS account.) You can also find your AZs by running through the Launch Instance wizard in the AWS GUI and seeing what zones it makes available in the dropdown.

We will be using 3 availability zones, as the cluster will be running 3 master nodes for HA quorum.

If you are deploying in us-east-1 or any other crowded region, it's best to avoid the "a" AZ. This example will use AZs "d", "e", and "f".

VPC CIDR

Kops is going to create a VPC for kubernetes to live in. This can be any valid private IP CIDR, but I tend to use a /16 in the 172 range, eg 172.20.0.0/16. I allocate large IP ranges because your future config may wish to assign an IP address per kubernetes pod, which can be thousands of IPs on a large product.

If you need to VPC peer with any other VPCs, make sure that this CIDR does not overlap any of them, and that it is also unique in the list of peers for your target. Eg, if your target already has a 172.20.0.0/16 peer, you cannot use that range for peering with your target.

This example will use 172.20.0.0/16.

EC2 Instance sizes

We will be using m4.large for the masters, which is considered sufficient to run a cluster of ~ 100 worker nodes.

We will use m4.2xlarge for our worker nodes.

Project setup

git

Most teams are using a single project-specific git repo, eg "ampersand-kubernetes", to store configs for all clusters pertaining to their project. The repo generally follows this structure:

/<environment>/terraform
/<environment>/<namespace>/
/<environment>/<namespace>/configmaps/
...

All of our example work here will be performed in the /dev-us-east-1/terraform directory, with exception of the .gitignore in the root directory.

.gitignore:

terraform.tfstate
terraform.tfstate.backup
.terraform/
.envrc

direnv

We will be using direnv to set some environment variables whenever you cd into the terraform directory. The pattern I use is to place the config that you wish to share with teammates into an envrc file (which is not read by direnv), and this is checked into git. Upon checkout of the repo, you cp envrc .envrc (note: .envrc is gitignored), and modify it to match your workstation setup. If you've never used direnv before, note that any time you edit your .envrc file, direnv will refuse to use it until you run direnv allow. If you never get prompted to run this when cded into the terraform directory, then direnv is not installed properly.

Create envrc:

PATH_add /path/to/terraform/0.11.7
PATH_add /path/to/kops/1.8.1
PATH_add /path/to/kubectl/1.10.2

export AWS_PROFILE=ampersand-nonprod
export AMPERSAND_DEV_K8_CLUSTER_NAME=ampersand-dev.bigmoney.mycompany.com
export AMPERSAND_DEV_K8_STATE_STORE=s3://ampersand-dev-remote-state/kops-infrastructure.tfstate
export KOPS_STATE_STORE="$AMPERSAND_DEV_K8_STATE_STORE" #for kops convenience

Now cp envrc .envrc and edit the PATHs to match your workstation, eg:

PATH_add /Users/bigshot/bin/terraform/0.11.7
PATH_add /Users/bigshot/bin/kops/1.8.1
PATH_add /Users/bigshot/bin/kubectl/1.10.2

export AWS_PROFILE=ampersand-nonprod
export AMPERSAND_DEV_K8_CLUSTER_NAME=ampersand-dev.bigmoney.mycompany.com
export AMPERSAND_DEV_K8_STATE_STORE=s3://ampersand-dev-remote-state/kops-infrastructure.tfstate
export KOPS_STATE_STORE="$AMPERSAND_DEV_K8_STATE_STORE" #for kops convenience

kops bootstrap

Create an S3 bucket

This will hold both your terraform remote state and your kops configs. Note: normally you would declare a provider in terraform to configure aws access, but kops is going to declare one in a minute and you can't have two, so we will just enter that information interactively until we get there.

terraform.tf:

terraform {
  required_version = ">= 0.11.7" # set this to your terraform version
}

variables.tf:

variable "product" {}

variable "tier" {}

variable "region" {}

terraform.tfvars:

product = "ampersand"
tier = "dev"
region = "us-east-1"

remote_state_bucket.tf:

resource "aws_s3_bucket" "ampersand_remote_state" {
  bucket = "${var.product}-${var.tier}-${var.region}-k8s-remote-state"
  versioning {
    enabled = true
  }
}

Run terraform init and terraform apply. You will be prompted for the aws region since you didn't declare the provider.

Create a Route53 domain for k8s

Add to variables.tf:

variable "route53_k8s_domain" {}

Add to terraform.tfvars:

route53_k8s_domain = "ampersand-dev.bigmoney.mycompany.com"

Create route53_k8s.tf:

resource "aws_route53_zone" "k8s" {
  name = "${var.route53_k8s_domain}"
}

Run terraform apply to create the domain.

If this is a subdomain, we now need to delegate access from the parent domain before the world can use it. The gist of this is: go look at your new subdomain in the AWS GUI, find the SOA record, and copy the list of AWS DNS servers there. Now go to your parent domain, create an NS record with the name of your subdomain (in this case, "ampersand-dev"), and paste the subdomain's DNS servers. If you're struggling, maybe see the aws docs for zone delegation.

Once you've created the delegation, you should wait 5-10 minutes before testing, as this process can take a few minutes to take effect, and some ISPs will cache the "domain not found" response, leading you to think that things are broken when they are not.

To verify that the subdomain is ready, add an A record in the GUI called test and point it to 4.4.4.4 or your favorite IP. The subdomain is working when you can nslookup test.ampersand-dev.bigmoney.mycompany.com from your workstation.

Enable terraform remote state

You now have some terraform state, and best practices are to store this in s3.

Update terraform.tf to add the backend:

terraform {
  required_version = ">= 0.11.7"
  backend "s3" {
    bucket = "ampersand-dev-remote-state"
    key    = "kops-infrastructure.tfstate"
    region = "us-east-1"
  }
}

Run terraform init.

Terraform will prompt for confirmation, and then upload your terraform.tfstate to s3 and zero out the local file. terraform apply should not show any changes needed. You can safely delete terraform.tfstate at this point, as well as terraform.tfstate.backup if it exists.

Create an SSH keypair

When kops creates your cluster, it will create a new public SSH key at AWS, with the contents that you specify at the command line. Run ssh-keygen to create a keypair locally, and for clarity rename the files to ampersand-dev-us-east-1.pub and ampersand-dev-us-east-1.pem. We'll reference the .pub in a moment.

Create your cluster

Make sure you are cded into the /dev/terraform directory of your kubernetes git repo, and verify that direnv has loaded your .envrc correctly and the environment variables are set. Then:

kops create cluster \
  --cloud=aws \
  --name=$AMPERSAND_DEV_K8_CLUSTER_NAME \
  --state=$AMPERSAND_DEV_K8_STATE_STORE \
  --dns-zone=$AMPERSAND_DEV_K8_CLUSTER_NAME \
  --zones us-east-1d,us-east-1e,us-east-1f \
  --master-zones us-east-1d,us-east-1e,us-east-1f \
  --network-cidr 172.20.0.0/16 \
  --topology private \
  --networking calico \
  --node-size m4.2xlarge \
  --master-size m4.large \
  --ssh-public-key=/path/to/ampersand-dev-us-east-1.pub \
  --target=terraform \
  --out=.

This command will 1) Create some kops config yaml in the s3 bucket, 2) generate some terraform locally based off of that config, and 3) update your kubectl config to create the new context, and switch to it. You can verify this with kubectl config current-context.

Now you can run terraform apply and it should create your cluster.

It will take several minutes for the nodes to come up, and several more for the nodes to start running the correct pods. You can now use kops validate cluster to see the status of your cluster. It will clearly tell you if any problems are found, eg nodes not coming up, or pods not launching correctly. Once the nodes are up, you can also start using kubectl get po -n kube-system to see the status of system pods.

Git commit!

Next steps

You now have an empty cluster with 2 worker nodes. For any future cluster changes, you will follow this pattern:

  1. kops edit cluster to change cluster-wide things like the k8s version, or kops edit ig nodes to edit an instancegroup (in this case, the "nodes" group, which is your default set of worker nodes). This will only edit the kops config files in s3.
  2. kops update cluster --out=. --target=terraform to generate new terraform config.
  3. terraform apply

For starters, you probably want more than 2 worker nodes, so perform the above with kops edit ig nodes and edit the minSize and maxSize.

Finally, you can let kops manage your IAM roles and some other details, but I like to keep all of the AWS stuff in terraform when possible and keep the kops config minimal. You can't edit kubernetes.tf because that will get overwritten every time you run kops update cluster, but you can create additional .tf files which reference the terraform resources created by kops. For example, I usually create a nodes-iam.tf file to control all of the IAM policies for worker nodes. Here is a sample file which will grant access to autoscaling, so that you can install cluster-autoscaler.

nodes-iam.tf:

### Grant all additional permissions for the nodes ig

resource "aws_iam_policy" "ampersand_dev_k8s_nodes_policy" {
    name        = "ampersand_dev_k8s_nodes_policy"
    description = "Ampersand Dev K8s nodes policy"
    policy = <<EOF
{
  "Version": "2012-10-17",
    {
      "Sid": "ampersandK8sNodesASGforClusterAutoscaling",
      "Effect": "Allow",
      "Action": [
          "autoscaling:DescribeAutoScalingGroups",
          "autoscaling:DescribeAutoScalingInstances",
          "autoscaling:DescribeLaunchConfigurations",
          "autoscaling:DescribeTags",
          "autoscaling:SetDesiredCapacity",
          "autoscaling:TerminateInstanceInAutoScalingGroup"
      ],
      "Resource": "*"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "ampersand_dev_k8s_nodes_policy" {
  role       = "${aws_iam_role.nodes-ampersand-dev-us-east-1-bigmoney-weather-com.name}"
  policy_arn = "${aws_iam_policy.ampersand_dev_k8s_nodes_policy.arn}"
}

And then terraform apply.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment