Configure EBS dynamic provisioning on an EKS Cluster
Background
I used pulumi to deploy an EKS cluster via: https://www.pulumi.com/templates/kubernetes/aws/ I noticed a gp2 storage class was defined after the cluster was up. I attempted to launch an application with a PVC expecting to see a PV be created but PVC was stuck in pending and I didn't know why.
Resolution
As of k8s 1.23, EKS needs the ebs-csi AddOn installed for dynamic provisioning of EBS Pulumi is not installing this automatically with an EKS cluster, there is an open issue tracking it: pulumi/pulumi-eks#833
Manual Steps
- Associate an IAM OIDC provider to the cluster
NAME="eks-cluster-eksCluster-cfb5b82"
REGION="us-west-2"
#https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html
eksctl utils associate-iam-oidc-provider --cluster $NAME --region $REGION --approve
- Create a role for the AddOn to use
NAME="eks-cluster-eksCluster-cfb5b82"
REGION="us-west-2"
eksctl create iamserviceaccount \
--name ebs-csi-controller-sa \
--namespace kube-system \
--cluster $NAME \
--attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
--approve \
--role-only \
--role-name JWM_AmazonEKS_EBS_CSI_DriverRole \
--region ${REGION}
- Example of a successful run
#$ ./create_ebs_csi_iam_role.sh
#2023-03-29 06:23:22 [ℹ] 1 iamserviceaccount (kube-system/ebs-csi-controller-sa) was included (based on the include/exclude rules)
#2023-03-29 06:23:22 [!] serviceaccounts in Kubernetes will not be created or modified, since the option --role-only is used
#2023-03-29 06:23:22 [ℹ] 1 task: { create IAM role for serviceaccount "kube-system/ebs-csi-controller-sa" }
#2023-03-29 06:23:22 [ℹ] building iamserviceaccount stack "eksctl-eks-cluster-eksCluster-cfb5b82-addon-iamserviceaccount-kube-system-ebs-csi-controller-sa"
#2023-03-29 06:23:22 [ℹ] deploying stack "eksctl-eks-cluster-eksCluster-cfb5b82-addon-iamserviceaccount-kube-system-ebs-csi-controller-sa"
#2023-03-29 06:23:22 [ℹ] waiting for CloudFormation stack "eksctl-eks-cluster-eksCluster-cfb5b82-addon-iamserviceaccount-kube-system-ebs-csi-controller-sa"
#2023-03-29 06:23:53 [ℹ] waiting for CloudFormation stack "eksctl-eks-cluster-eksCluster-cfb5b82-addon-iamserviceaccount-kube-system-ebs-csi-controller-sa"
-
Install the AddOn: visit: https://us-west-2.console.aws.amazon.com/eks/home?region=us-west-2#/clusters/eks-cluster-eksCluster-cfb5b82/create-add-ons Select the IAM Role we just created: JWM_AmazonEKS_EBS_CSI_DriverRole
-
Verify it is installed:
aws-node-8hf7g 1/1 Running 0 11h
aws-node-bl4pk 1/1 Running 0 11h
aws-node-fxf4n 1/1 Running 0 11h
coredns-67f8f59c6c-dl45c 1/1 Running 0 11h
coredns-67f8f59c6c-z7z4x 1/1 Running 0 11h
ebs-csi-controller-6878c4b844-jddkf 6/6 Running 0 7m11s
ebs-csi-controller-6878c4b844-l9pcs 6/6 Running 0 7m11s
ebs-csi-node-st9kd 3/3 Running 0 7m12s
ebs-csi-node-vjftq 3/3 Running 0 7m12s
ebs-csi-node-xzk9v 3/3 Running 0 7m12s
kube-proxy-g4g6z 1/1 Running 0 11h
kube-proxy-hsrvb 1/1 Running 0 11h
kube-proxy-l45bh 1/1 Running 0 11h
metrics-server-55c774cdbb-fwhmm 1/1 Running 0 11h
- Note the
ebs-csi-*pods