Exposing Services on EKS

EXPOSING_SERVICES_ON_EKS.MD

Exposing Kubernetes Applications, Part 2: AWS Load Balancer Controller

Prerequisites

Create AWS Load Balancer Controller Identity and Access Management (AWS IAM) Policy

Download IAM policy for the AWS Load Balancer Controller

• curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.6/docs/install/iam_policy.json

Create an IAM policy called AWSLoadBalancerControllerIAMPolicy

• aws iam create-policy \
  --policy-name AWSLoadBalancerControllerIAMPolicy \
  --policy-document file://iam-policy.json

Determine whether you have existing AWSLoadBalancerControllerIAMPolicy

• aws iam list-policies | grep AWSLoadBalancerControllerIAMPolicy

Create Cluster

Note that OIDC IAM provider registration and AWS Load Balancer Controller service account creation are done automatically by eksctl based on the configuration below and do not need to be explicitly handled.

cat <<EOF> config-gitignore.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: aws-load-balancer-controller-walkthrough
  region: ${AWS_REGION}
  version: '1.23'
vpc:
  id: '${VPC}'
  subnets:
    private:
      pvt-1:
            id: '${PVT_SUB1}'
      pvt-2:
            id: '${PVT_SUB2}'
iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: aws-load-balancer-controller
        namespace: kube-system
      attachPolicyARNs:
        - arn:aws:iam::${AWS_ACCOUNT_ID}:policy/AWSLoadBalancerControllerIAMPolicy
managedNodeGroups:
  - name: main-ng
    instanceType: m5.large
    desiredCapacity: 1
    privateNetworking: true
    subnets:
      - pvt-1
      - pvt-2
EOF

envsubst < config-gitignore.yaml | eksctl create cluster -f -

eksctl delete cluster --region=us-east-1 --name=aws-load-balancer-controller-walkthrough

Install Helm

Install the Controller Using helm

helm repo add eks https://aws.github.io/eks-charts

helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller \
    --namespace kube-system \
    --set clusterName=aws-load-balancer-controller-walkthrough \
    --set serviceAccount.create=false \
    --set serviceAccount.name=aws-load-balancer-controller

kubectl -n kube-system rollout status deployment aws-load-balancer-controller

kubectl get deployment -n kube-system aws-load-balancer-controller

Deploy the Testing Services

Create the Service Manifest File

cat <<'EOF'> service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${SERVICE_NAME}
  namespace: ${NS}
  labels:
    app.kubernetes.io/name: ${SERVICE_NAME}
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ${SERVICE_NAME}
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ${SERVICE_NAME}
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: ${SERVICE_NAME}
          image: hashicorp/http-echo
          imagePullPolicy: IfNotPresent
          args:
            - -listen=:3000
            - -text=${SERVICE_NAME}
          ports:
            - name: app-port
              containerPort: 3000
          resources:
            requests:
              cpu: 0.125
              memory: 50Mi
---
apiVersion: v1
kind: Service
metadata:
  name: ${SERVICE_NAME}
  namespace: ${NS}
  labels:
    app.kubernetes.io/name: ${SERVICE_NAME}
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: ${SERVICE_NAME}
  ports:
    - name: svc-port
      port: 80
      targetPort: app-port
      protocol: TCP
EOF

Deploy and Verify the Services

kubectl create namespace apps

SERVICE_NAME=first NS=apps envsubst < service.yaml | kubectl apply -f -
SERVICE_NAME=second NS=apps envsubst < service.yaml | kubectl apply -f -
SERVICE_NAME=third NS=apps envsubst < service.yaml | kubectl apply -f -
SERVICE_NAME=fourth NS=apps envsubst < service.yaml | kubectl apply -f -
SERVICE_NAME=error NS=apps envsubst < service.yaml | kubectl apply -f -

Verify that all the resources are deployed:

kubectl get pod,svc -n apps

Create the Ingress Manifest file

cat <<'EOF'> ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ${NS}-ingress
  namespace: ${NS}
  annotations:
    alb.ingress.kubernetes.io/load-balancer-name: ${NS}-ingress
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
spec:
  ingressClassName: alb
#  defaultBackend:
#    service:
#      name: error
#      port:
#        name: svc-port
  rules:
    - http:
        paths:
          - path: /first
            pathType: Prefix
            backend:
              service:
                name: first
                port:
                  name: svc-port
          - path: /second
            pathType: Prefix
            backend:
              service:
                name: second
                port:
                  name: svc-port
EOF

Deploy the Ingress Resource

NS=apps envsubst < ingress.yaml | kubectl apply -f -

kubectl get ingress -n apps

Hostname-Based routing

Alter the Ingress Definition

cat <<'EOF'> ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ${NS}-ingress
  namespace: ${NS}
  annotations:
    alb.ingress.kubernetes.io/load-balancer-name: ${NS}-ingress
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
spec:
  ingressClassName: alb
  defaultBackend:
    service:
      name: error
      port:
        name: svc-port
  rules:
    - host: a.example.com
      http:
        paths:
          - path: /first
            pathType: Prefix
            backend:
              service:
                name: first
                port:
                  name: svc-port
          - path: /second
            pathType: Prefix
            backend:
              service:
                name: second
                port:
                  name: svc-port
    - host: '*.example.com'
      http:
        paths:
          - path: /third
            pathType: Prefix
            backend:
              service:
                name: third
                port:
                  name: svc-port
          - path: /fourth
            pathType: Prefix
            backend:
              service:
                name: fourth
                port:
                  name: svc-port
EOF

Deploy the updated Ingress

NS=apps envsubst < ingress.yaml | kubectl apply -f -

Exposing Kubernetes Applications, Part 3: NGINX Ingress Controller

NGINX Ingress Controller

Deploy app

SERVICE_NAME=another-error NS=apps envsubst < service.yaml | kubectl apply -f -

kubectl get pod,svc -n apps

Install the NGINX Ingress Controller

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

helm upgrade -i ingress-nginx ingress-nginx/ingress-nginx \
    --version 4.2.3 \
    --namespace kube-system \
    --set controller.service.type=ClusterIP

kubectl -n kube-system rollout status deployment ingress-nginx-controller

kubectl get deployment -n kube-system ingress-nginx-controller

Create the Ingress Manifest file and Deploy the Ingress

cat <<'EOF'> nginx-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ${NS}-nginx-ingress
  namespace: ${NS}
spec:
  ingressClassName: nginx
#  defaultBackend:
#    service:
#      name: error
#      port:
#        name: svc-port
  rules:
    - http:
        paths:
          - path: /first
            pathType: Prefix
            backend:
              service:
                name: first
                port:
                  name: svc-port
          - path: /second
            pathType: Prefix
            backend:
              service:
                name: second
                port:
                  name: svc-port
EOF

NS=apps envsubst < nginx-ingress.yaml | kubectl apply -f -

kubectl port-forward -n kube-system svc/ingress-nginx-controller 8080:80

Host-based Routing

Update and Deploy the Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ${NS}-nginx-ingress
  namespace: ${NS}
spec:
  ingressClassName: nginx
  defaultBackend:
    service:
      name: error
      port:
        name: svc-port
  rules:
    - host: a.example.com
      http:
        paths:
          - path: /first
            pathType: Prefix
            backend:
              service:
                name: first
                port:
                  name: svc-port
    - host: b.example.com
      http:
        paths:
          - path: /second
            pathType: Prefix
            backend:
              service:
                name: second
                port:
                  name: svc-port

NS=apps envsubst < nginx-ingress.yaml | kubectl apply -f -

Provision AWS NLB for the NGINX Ingress Controller

Create AWS Load Balancer Controller Identity and Access Management (AWS IAM) Policy

Create a Service Account for AWS Load Balancer Controller

Install the CRDs

kubectl apply -f https://raw.githubusercontent.com/aws/eks-charts/master/stable/aws-load-balancer-controller/crds/crds.yaml

Install the aws-load-balancer-controller Using Helm

Redeploy NGINX Ingress Controller

cat <<EOF> values.yaml
controller:
  service:
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-name: apps-nginx-ingress
      service.beta.kubernetes.io/aws-load-balancer-type: external
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: http
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /healthz
      service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: 10254
EOF

helm upgrade -i ingress-nginx ingress-nginx/ingress-nginx \
    --version 4.2.3 \
    --namespace kube-system \
    --values values.yaml
    
kubectl -n kube-system rollout status deployment ingress-nginx-controller

kubectl get deployment -n kube-system ingress-nginx-controller

References:

• https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/
• https://kubernetes.github.io/ingress-nginx/
• https://github.com/aws/eks-charts
• https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html
  • https://www.youtube.com/watch?v=iBVaRZU8ALc