TF kubernetes cluster creation

README.md

Comparing k8s cluster creation under AWS and GCP with use of the kubernetes provider.

AWS EKS

Kubernetes provider consistently fails on first access. Service is returned as ready before the endpoint is actually usable. Retries or sleeping "long enough" required.

GCP GKE

Cluster ready status is reported when the cluster is actually ready to be used. Chaining of the kubernetes provider does not present problems in test runs done.

Usage

If you actually want to try this yourself you need to move the 3 TF files into subfolders with matching names:

• aws.tf -> aws/aws.tf
• gcp.tf -> gcp/gcp.tf
• kubernetes.tf -> kubernetes/kubernetes.tf

aws-output.txt

module.vpc.aws_vpc.this[0]: Creating...
aws_iam_role.cluster: Creating...
aws_iam_role.cluster: Creation complete after 1s [id=test-cluster-20191214165125996900000001]
aws_iam_role_policy_attachment.cluster_AmazonEKSServicePolicy: Creating...
aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy: Creating...
aws_iam_role_policy_attachment.cluster_AmazonEKSServicePolicy: Creation complete after 1s [id=test-cluster-20191214165125996900000001-20191214165127676000000002]
aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy: Creation complete after 1s [id=test-cluster-20191214165125996900000001-20191214165127709400000003]
module.vpc.aws_vpc.this[0]: Creation complete after 6s [id=vpc-02d8caf62ac5cb7bd]
module.vpc.aws_internet_gateway.this[0]: Creating...
module.vpc.aws_route_table.public[0]: Creating...
module.vpc.aws_subnet.public[0]: Creating...
module.vpc.aws_subnet.public[1]: Creating...
module.vpc.aws_route_table.public[0]: Creation complete after 2s [id=rtb-033f7340fb894f278]
module.vpc.aws_subnet.public[0]: Creation complete after 3s [id=subnet-00e9c8ab65bac5302]
module.vpc.aws_subnet.public[1]: Creation complete after 3s [id=subnet-06d7ef7d09b7d4694]
module.vpc.aws_route_table_association.public[0]: Creating...
module.vpc.aws_route_table_association.public[1]: Creating...
aws_eks_cluster.this: Creating...
module.vpc.aws_internet_gateway.this[0]: Creation complete after 3s [id=igw-0417d19bb014b6b26]
module.vpc.aws_route.public_internet_gateway[0]: Creating...
module.vpc.aws_route_table_association.public[0]: Creation complete after 0s [id=rtbassoc-00ae7485b970b1fa9]
module.vpc.aws_route_table_association.public[1]: Creation complete after 0s [id=rtbassoc-049f11e88e2955e9a]
module.vpc.aws_route.public_internet_gateway[0]: Creation complete after 1s [id=r-rtb-033f7340fb894f2781080289494]
aws_eks_cluster.this: Still creating... [10s elapsed]
aws_eks_cluster.this: Still creating... [20s elapsed]
aws_eks_cluster.this: Still creating... [30s elapsed]
aws_eks_cluster.this: Still creating... [40s elapsed]
aws_eks_cluster.this: Still creating... [50s elapsed]
aws_eks_cluster.this: Still creating... [1m0s elapsed]
aws_eks_cluster.this: Still creating... [1m10s elapsed]
aws_eks_cluster.this: Still creating... [1m20s elapsed]
aws_eks_cluster.this: Still creating... [1m30s elapsed]
aws_eks_cluster.this: Still creating... [1m40s elapsed]
aws_eks_cluster.this: Still creating... [1m50s elapsed]
aws_eks_cluster.this: Still creating... [2m0s elapsed]
aws_eks_cluster.this: Still creating... [2m10s elapsed]
aws_eks_cluster.this: Still creating... [2m20s elapsed]
aws_eks_cluster.this: Still creating... [2m30s elapsed]
aws_eks_cluster.this: Still creating... [2m40s elapsed]
aws_eks_cluster.this: Still creating... [2m50s elapsed]
aws_eks_cluster.this: Still creating... [3m0s elapsed]
aws_eks_cluster.this: Still creating... [3m10s elapsed]
aws_eks_cluster.this: Still creating... [3m20s elapsed]
aws_eks_cluster.this: Still creating... [3m30s elapsed]
aws_eks_cluster.this: Still creating... [3m40s elapsed]
aws_eks_cluster.this: Still creating... [3m50s elapsed]
aws_eks_cluster.this: Still creating... [4m0s elapsed]
aws_eks_cluster.this: Still creating... [4m10s elapsed]
aws_eks_cluster.this: Still creating... [4m20s elapsed]
aws_eks_cluster.this: Still creating... [4m30s elapsed]
aws_eks_cluster.this: Still creating... [4m40s elapsed]
aws_eks_cluster.this: Still creating... [4m50s elapsed]
aws_eks_cluster.this: Still creating... [5m0s elapsed]
aws_eks_cluster.this: Still creating... [5m10s elapsed]
aws_eks_cluster.this: Still creating... [5m20s elapsed]
aws_eks_cluster.this: Still creating... [5m30s elapsed]
aws_eks_cluster.this: Still creating... [5m40s elapsed]
aws_eks_cluster.this: Still creating... [5m50s elapsed]
aws_eks_cluster.this: Still creating... [6m0s elapsed]
aws_eks_cluster.this: Still creating... [6m10s elapsed]
aws_eks_cluster.this: Still creating... [6m20s elapsed]
aws_eks_cluster.this: Still creating... [6m30s elapsed]
aws_eks_cluster.this: Still creating... [6m40s elapsed]
aws_eks_cluster.this: Still creating... [6m50s elapsed]
aws_eks_cluster.this: Still creating... [7m0s elapsed]
aws_eks_cluster.this: Still creating... [7m10s elapsed]
aws_eks_cluster.this: Still creating... [7m20s elapsed]
aws_eks_cluster.this: Still creating... [7m30s elapsed]
aws_eks_cluster.this: Still creating... [7m40s elapsed]
aws_eks_cluster.this: Still creating... [7m50s elapsed]
aws_eks_cluster.this: Still creating... [8m0s elapsed]
aws_eks_cluster.this: Still creating... [8m10s elapsed]
aws_eks_cluster.this: Still creating... [8m20s elapsed]
aws_eks_cluster.this: Still creating... [8m30s elapsed]
aws_eks_cluster.this: Still creating... [8m40s elapsed]
aws_eks_cluster.this: Still creating... [8m50s elapsed]
aws_eks_cluster.this: Still creating... [9m0s elapsed]
aws_eks_cluster.this: Still creating... [9m10s elapsed]
aws_eks_cluster.this: Still creating... [9m20s elapsed]
aws_eks_cluster.this: Still creating... [9m30s elapsed]
aws_eks_cluster.this: Still creating... [9m40s elapsed]
aws_eks_cluster.this: Still creating... [9m50s elapsed]
aws_eks_cluster.this: Still creating... [10m0s elapsed]
aws_eks_cluster.this: Still creating... [10m10s elapsed]
aws_eks_cluster.this: Still creating... [10m20s elapsed]
aws_eks_cluster.this: Still creating... [10m30s elapsed]
aws_eks_cluster.this: Still creating... [10m40s elapsed]
aws_eks_cluster.this: Still creating... [10m50s elapsed]
aws_eks_cluster.this: Creation complete after 10m56s [id=test]
data.aws_eks_cluster_auth.this: Refreshing state...
data.aws_eks_cluster.this: Refreshing state...
module.kubernetes.kubernetes_config_map.test: Creating...
module.kubernetes.kubernetes_config_map.test: Still creating... [10s elapsed]
module.kubernetes.kubernetes_config_map.test: Still creating... [20s elapsed]
module.kubernetes.kubernetes_config_map.test: Still creating... [30s elapsed]

Error: Post https://471AAEA997AF84FF202544383113B89F.yl4.us-east-1.eks.amazonaws.com/api/v1/namespaces/default/configmaps: dial tcp 3.232.192.19:443: i/o timeout

  on ../kubernetes/main.tf line 13, in resource "kubernetes_config_map" "test":
  13: resource "kubernetes_config_map" "test" {

aws.tf

# AWS
variable "region" {
  default = "us-east-1"
}
variable "azs" {
  default = ["us-east-1a", "us-east-1b"]
}
variable "cluster_name" {
  default = "test"
}
variable "cidr" {
  default = "10.0.0.0/16"
}

provider "aws" {
  region              = var.region
  version             = "~> 2.40"
  ignore_tag_prefixes = ["kubernetes.io/cluster/"]
}

data "aws_availability_zones" "zones" {}

module "vpc" {
  source             = "terraform-aws-modules/vpc/aws"
  version            = "~> 2.0"
  name               = var.cluster_name
  cidr               = var.cidr
  public_subnets     = cidrsubnets(var.cidr, 4, 4)
  enable_nat_gateway = false
  azs                = var.azs
}

data "aws_iam_policy_document" "cluster_assume_role" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["eks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "cluster" {
  name_prefix        = "${var.cluster_name}-cluster-"
  assume_role_policy = data.aws_iam_policy_document.cluster_assume_role.json
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.cluster.name
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSServicePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  role       = aws_iam_role.cluster.name
}

resource "aws_eks_cluster" "this" {
  name     = var.cluster_name
  role_arn = aws_iam_role.cluster.arn
  version  = "1.14"

  vpc_config {
    subnet_ids = module.vpc.public_subnets
  }

  depends_on = [
    aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy,
    aws_iam_role_policy_attachment.cluster_AmazonEKSServicePolicy,
  ]
}

# Kubernetes
data "aws_eks_cluster_auth" "this" {
  name = aws_eks_cluster.this.id
}

module "kubernetes" {
  source                 = "../kubernetes"
  host                   = aws_eks_cluster.this.endpoint
  cluster_ca_certificate = aws_eks_cluster.this.certificate_authority.0.data
  token                  = data.aws_eks_cluster_auth.this.token
}

gcp-output.txt

google_container_cluster.this: Creating...
google_container_cluster.this: Still creating... [10s elapsed]
google_container_cluster.this: Still creating... [20s elapsed]
google_container_cluster.this: Still creating... [30s elapsed]
google_container_cluster.this: Still creating... [40s elapsed]
google_container_cluster.this: Still creating... [50s elapsed]
google_container_cluster.this: Still creating... [1m0s elapsed]
google_container_cluster.this: Still creating... [1m10s elapsed]
google_container_cluster.this: Still creating... [1m20s elapsed]
google_container_cluster.this: Still creating... [1m30s elapsed]
google_container_cluster.this: Still creating... [1m40s elapsed]
google_container_cluster.this: Still creating... [1m50s elapsed]
google_container_cluster.this: Still creating... [2m0s elapsed]
google_container_cluster.this: Still creating... [2m10s elapsed]
google_container_cluster.this: Still creating... [2m20s elapsed]
google_container_cluster.this: Still creating... [2m30s elapsed]
google_container_cluster.this: Still creating... [2m40s elapsed]
google_container_cluster.this: Still creating... [2m50s elapsed]
google_container_cluster.this: Still creating... [3m0s elapsed]
google_container_cluster.this: Still creating... [3m10s elapsed]
google_container_cluster.this: Still creating... [3m20s elapsed]
google_container_cluster.this: Still creating... [3m30s elapsed]
google_container_cluster.this: Still creating... [3m40s elapsed]
google_container_cluster.this: Still creating... [3m50s elapsed]
google_container_cluster.this: Still creating... [4m0s elapsed]
google_container_cluster.this: Still creating... [4m10s elapsed]
google_container_cluster.this: Still creating... [4m20s elapsed]
google_container_cluster.this: Creation complete after 4m27s [id=projects/test/locations/us-central1-a/clusters/test]
module.kubernetes.kubernetes_config_map.test: Creating...
module.kubernetes.kubernetes_config_map.test: Creation complete after 1s [id=default/test]

gcp.tf

variable "project" {}
variable "region" {
  default = "us-central1"
}
variable "cluster_name" {
  default = "test"
}

provider "google" {
  project = var.project
  version = "~> 3.2"
  region  = var.region
}

resource "google_container_cluster" "this" {
  name               = var.cluster_name
  location           = "us-central1-a"
  initial_node_count = 1

  master_auth {
    username = ""
    password = ""

    client_certificate_config {
      issue_client_certificate = false
    }
  }
}

# Kubernetes
data "google_client_config" "default" {}

module "kubernetes" {
  source                 = "../kubernetes"
  host                   = "https://${google_container_cluster.this.endpoint}"
  cluster_ca_certificate = google_container_cluster.this.master_auth.0.cluster_ca_certificate
  token                  = data.google_client_config.default.access_token
}

kubernetes.tf

variable "host" {}
variable "cluster_ca_certificate" {}
variable "token" {}

provider "kubernetes" {
  host                   = var.host
  cluster_ca_certificate = base64decode(var.cluster_ca_certificate)
  token                  = var.token
  load_config_file       = false
  version                = "~> 1.10"
}

resource "kubernetes_config_map" "test" {
  metadata {
    name = "test"
  }

  data = {
    test = "testdata"
  }
}