Skip to content

Instantly share code, notes, and snippets.

@arun-gupta
Last active December 8, 2021 05:04
Show Gist options
  • Save arun-gupta/02f534c7720c8e9c9a875681b430441a to your computer and use it in GitHub Desktop.
Save arun-gupta/02f534c7720c8e9c9a875681b430441a to your computer and use it in GitHub Desktop.
Federated Kubernetes Cluster using Kops on AWS

Let’s set up a federation between earth and mars Kubernetes cluster on AWS.

Earth cluster

  1. Create hosted zone:

    ID=$(uuidgen) && \
       aws route53 create-hosted-zone \
       --name earth.kubernetes-aws.io \
       --caller-reference $ID \
       | jq .DelegationSet.NameServers
  2. Setup NS records in DNS provider. Make sure dig earth.kubernetes-aws.io NS return NS records:

    dig earth.kubernetes-aws.io NS
    
    ; <<>> DiG 9.8.3-P1 <<>> earth.kubernetes-aws.io NS
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16919
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4
    
    ;; QUESTION SECTION:
    ;earth.kubernetes-aws.io.	IN	NS
    
    ;; ANSWER SECTION:
    earth.kubernetes-aws.io. 3271	IN	NS	ns-752.awsdns-30.net.
    earth.kubernetes-aws.io. 3271	IN	NS	ns-1196.awsdns-21.org.
    earth.kubernetes-aws.io. 3271	IN	NS	ns-1799.awsdns-32.co.uk.
    earth.kubernetes-aws.io. 3271	IN	NS	ns-436.awsdns-54.com.
    
    ;; ADDITIONAL SECTION:
    ns-436.awsdns-54.com.	162638	IN	A	205.251.193.180
    ns-752.awsdns-30.net.	161105	IN	A	205.251.194.240
    ns-1196.awsdns-21.org.	161034	IN	A	205.251.196.172
    ns-1799.awsdns-32.co.uk. 148088	IN	A	205.251.199.7
    
    ;; Query time: 13 msec
    ;; SERVER: 10.4.4.10#53(10.4.4.10)
    ;; WHEN: Fri Aug 25 14:44:22 2017
    ;; MSG SIZE  rcvd: 245
  3. Create cluster:

    kops create cluster \
      --name=earth.kubernetes-aws.io \
      --master-count=3 \
      --master-zones us-east-1a,us-east-1b,us-east-1c \
      --node-count=3 \
      --zones=us-east-1a,us-east-1b,us-east-1c \
      --authorization=rbac \
      --state=s3://kubernetes-aws-io \
      --kubernetes-version=1.7.0 \
      --yes
  4. Validate cluster:

    kops validate cluster earth.kubernetes-aws.io --state=s3://kubernetes-aws-io
    Validating cluster earth.kubernetes-aws.io
    
    INSTANCE GROUPS
    NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
    master-us-east-1a	Master	m3.medium	1	1	us-east-1a
    master-us-east-1b	Master	m3.medium	1	1	us-east-1b
    master-us-east-1c	Master	c4.large	1	1	us-east-1c
    nodes			Node	t2.medium	3	3	us-east-1a,us-east-1b,us-east-1c
    
    NODE STATUS
    NAME				ROLE	READY
    ip-172-20-117-87.ec2.internal	node	True
    ip-172-20-125-87.ec2.internal	master	True
    ip-172-20-45-153.ec2.internal	master	True
    ip-172-20-55-15.ec2.internal	node	True
    ip-172-20-64-27.ec2.internal	master	True
    ip-172-20-76-15.ec2.internal	node	True
    
    Your cluster earth.kubernetes-aws.io is ready
  5. Set alias for context:

    kubectl config set-context earth --cluster=earth.kubernetes-aws.io --user=earth.kubernetes-aws.io
    Context "earth" created.
  6. Get nodes:

    kubectl --context=earth get nodes
    NAME                            STATUS    AGE       VERSION
    ip-172-20-117-87.ec2.internal   Ready     6m        v1.7.0
    ip-172-20-125-87.ec2.internal   Ready     7m        v1.7.0
    ip-172-20-45-153.ec2.internal   Ready     8m        v1.7.0
    ip-172-20-55-15.ec2.internal    Ready     6m        v1.7.0
    ip-172-20-64-27.ec2.internal    Ready     7m        v1.7.0
    ip-172-20-76-15.ec2.internal    Ready     6m        v1.7.0

Mars cluster

  1. Create hosted zone:

    ID=$(uuidgen) && \
       aws route53 create-hosted-zone \
       --name mars.kubernetes-aws.io \
       --caller-reference $ID \
       | jq .DelegationSet.NameServers
  2. Setup NS records in DNS provider. Make sure dig mars.kubernetes-aws.io NS return NS records:

    dig mars.kubernetes-aws.io NS
    
    ; <<>> DiG 9.8.3-P1 <<>> mars.kubernetes-aws.io NS
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64774
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4
    
    ;; QUESTION SECTION:
    ;mars.kubernetes-aws.io.		IN	NS
    
    ;; ANSWER SECTION:
    mars.kubernetes-aws.io.	2630	IN	NS	ns-26.awsdns-03.com.
    mars.kubernetes-aws.io.	2630	IN	NS	ns-964.awsdns-56.net.
    mars.kubernetes-aws.io.	2630	IN	NS	ns-1052.awsdns-03.org.
    mars.kubernetes-aws.io.	2630	IN	NS	ns-1965.awsdns-53.co.uk.
    
    ;; ADDITIONAL SECTION:
    ns-26.awsdns-03.com.	143078	IN	A	205.251.192.26
    ns-964.awsdns-56.net.	157684	IN	A	205.251.195.196
    ns-1052.awsdns-03.org.	23776	IN	A	205.251.196.28
    ns-1965.awsdns-53.co.uk. 75558	IN	A	205.251.199.173
    
    ;; Query time: 15 msec
    ;; SERVER: 10.4.4.10#53(10.4.4.10)
    ;; WHEN: Fri Aug 25 14:44:43 2017
    ;; MSG SIZE  rcvd: 243
  3. Create cluster

    kops create cluster \
      --name=mars.kubernetes-aws.io \
      --master-count=3 \
      --master-zones us-east-2a,us-east-2b,us-east-2c \
      --node-count=3 \
      --zones=us-east-2a,us-east-2b,us-east-2c \
      --authorization=rbac \
      --state=s3://kubernetes-aws-io \
      --kubernetes-version=1.7.0 \
      --yes
  4. Validate

    kops validate cluster mars.kubernetes-aws.io --state=s3://kubernetes-aws-io
    Validating cluster mars.kubernetes-aws.io
    
    INSTANCE GROUPS
    NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
    master-us-east-2a	Master	c4.large	1	1	us-east-2a
    master-us-east-2b	Master	c4.large	1	1	us-east-2b
    master-us-east-2c	Master	c4.large	1	1	us-east-2c
    nodes			Node	t2.medium	3	3	us-east-2a,us-east-2b,us-east-2c
    
    NODE STATUS
    NAME						ROLE	READY
    ip-172-20-107-105.us-east-2.compute.internal	node	True
    ip-172-20-126-49.us-east-2.compute.internal	master	True
    ip-172-20-41-181.us-east-2.compute.internal	node	True
    ip-172-20-62-64.us-east-2.compute.internal	master	True
    ip-172-20-89-187.us-east-2.compute.internal	node	True
    ip-172-20-89-96.us-east-2.compute.internal	master	True
    
    Your cluster mars.kubernetes-aws.io is ready
  5. Set alias for context:

    kubectl config set-context mars --cluster=mars.kubernetes-aws.io --user=mars.kubernetes-aws.io
    Context "mars" modified.
  6. Get nodes:

    kubectl --context=mars get nodes
    NAME                                           STATUS    AGE       VERSION
    ip-172-20-107-105.us-east-2.compute.internal   Ready     9m        v1.7.0
    ip-172-20-126-49.us-east-2.compute.internal    Ready     10m       v1.7.0
    ip-172-20-41-181.us-east-2.compute.internal    Ready     9m        v1.7.0
    ip-172-20-62-64.us-east-2.compute.internal     Ready     10m       v1.7.0
    ip-172-20-89-187.us-east-2.compute.internal    Ready     9m        v1.7.0
    ip-172-20-89-96.us-east-2.compute.internal     Ready     10m       v1.7.0

Setup federation

  1. Download k8s client binary:

    curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/kubernetes-client-darwin-amd64.tar.gz
    tar xzvf kubernetes-client-darwin-amd64.tar.gz
  2. Check context:

    kubectl config get-contexts
    CURRENT   NAME                      CLUSTER                   AUTHINFO                  NAMESPACE
              earth.kubernetes-aws.io   earth.kubernetes-aws.io   earth.kubernetes-aws.io
              mars                      mars.kubernetes-aws.io    mars.kubernetes-aws.io
    *         mars.kubernetes-aws.io    mars.kubernetes-aws.io    mars.kubernetes-aws.io
              earth                     earth.kubernetes-aws.io   earth.kubernetes-aws.io
  3. Use earth as host cluster:

    kubectl config use-context earth
    Switched to context "earth".
  4. Create RBAC role binding:

    kubectl create \
       clusterrolebinding \
       admin-to-cluster-admin-binding \
       --clusterrole=cluster-admin \
       --user=admin
  5. Deploy the federation control plane in host cluster:

    kubefed \
        init \
        interstellar \
        --host-cluster-context=earth \
        --dns-provider=aws-route53 \
        --dns-zone-name=kubernetes-aws.io

    Shows the output:

    Creating a namespace federation-system for federation system components... done
    Creating federation control plane service..... done
    Creating federation control plane objects (credentials, persistent volume claim)... done
    Creating federation component deployments... done
    Updating kubeconfig... done
    Waiting for federation control plane to come up.......................................................... done
    Federation API server is running at: aadbdf9358d1411e7a4ba0a49363c58c-672820986.us-east-1.elb.amazonaws.com
  6. Get contexts again to see the newly created namespace:

    kubectl config get-contexts
    CURRENT   NAME                      CLUSTER                   AUTHINFO                  NAMESPACE
              mars                      mars.kubernetes-aws.io    mars.kubernetes-aws.io
              mars.kubernetes-aws.io    mars.kubernetes-aws.io    mars.kubernetes-aws.io
    *         earth                     earth.kubernetes-aws.io   earth.kubernetes-aws.io
              earth.kubernetes-aws.io   earth.kubernetes-aws.io   earth.kubernetes-aws.io
              interstellar              interstellar              interstellar
  7. Change the context to federation context:

    kubectl config use-context interstellar
  8. Join earth and mars cluster to the federation:

    kubefed join \
       earth \
       --host-cluster-context=earth \
       --cluster-context=earth
    kubefed join \
       mars \
       --host-cluster-context=earth \
       --cluster-context=mars

    --cluster-context defaults to cluster name

  9. Check status of the clusters in the federation:

    kubectl --context=interstellar get clusters
    NAME      STATUS    AGE
    earth     Ready     4h
    mars      Ready     3h

Cleanup

  1. Unjoin earth and mars cluster from federation:

    kubefed unjoin earth --cluster-context=earth
    kubefed unjoin mars --cluster-context=earth
  2. Delete clusters

    kops delete cluster --name=earth.kubernetes-aws.io --yes
    kops delete cluster --name=mars.kubernetes-aws.io --yes

Misc

  1. Use aws ec2 describe-availability-zones --region=us-east-1 to find out a region with 3+ AZ

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment