Using Amazon CNI with kops-created Kubernetes cluster

readme.adoc

AWS CNI plugin is now merged with kops: kubernetes/kops#3997. This gist explains how to build kops, create a Kubernetes cluster using correct --networking option, and then test it.

Build kops

export GOPATH=`pwd`
mkdir src/k8s.io; cd src/k8s.io
git clone git@github.com:kubernetes/kops.git
cd kops
export S3_BUCKET_NAME=<some bucket you own>
export KOPS_STATE_STORE=s3://${S3_BUCKET_NAME}
export KOPS_BASE_URL=https://${S3_BUCKET_NAME}.s3.amazonaws.com/kops/dev/
make kops-install upload S3_BUCKET=s3://${S3_BUCKET_NAME} VERSION=dev

Create cluster

./.build/upload/kops/dev/darwin/amd64/kops create cluster \
--name example.cluster.k8s.local \
--zones us-east-1a,us-east-1b,us-east-1c \
--networking amazon-vpc-routed-eni \
--kubernetes-version 1.8.4 \
--yes

Create Deployment

ENIs and secondary IP addresses per instance type: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

One secondary IP address per ENI is reserved for gateway. All others are available to be assigned to the pod. So the total number of IP addresses available:

number of worker nodes * number of ENIs per instance type * (number of IP addresses allowed per ENI - 1)

Lets deploy:

1. Use deployment.yaml:

   apiVersion: extensions/v1beta1
   kind: Deployment
   metadata:
     name: nginx-deployment
   spec:
     replicas: 3
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - name: nginx
           image: nginx:1.12.1
           ports:
           - containerPort: 80
           - containerPort: 443

2. kubectl apply -f deployment.yaml

3. Scale replicas: kubectl scale --replicas=30 deployment/nginx-deployemnt

t2.medium allows 3 ENIs and 6 IP addresses. Two worker nodes are created. This means a total of 2 * 3 * (6 - 1), or 30. Scaling replicas beyond that, for this instance type, will show the additional pods as unavailable.

Only 27 pods are available instead of 30. Filed aws/amazon-vpc-cni-k8s#18.

Delete cluster

kops delete cluster --name example.cluster.k8s.local --yes

Debug

1. Log in to worker nodes

2. Run sudo /opt/cni/bin/aws-cni-support.sh

3. Share the log file: scp -i ~/.ssh/arun-us-east1.pem admin@<ip>:/var/log/aws-routed-eni/aws-cni-support.tar.gz aws-cni-support.tar.gz