eks-pod-increase.md

Summary

Pod hpa would create extra pods during deployment rolling update.

See kubernetes/kubernetes#72775

Environment

• Amazon EKS, kubernetes version 1.14.6

Reproduction Steps

Step1

Prepare the following test.yaml and apply it.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test
  name: test
  namespace: default
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
      annotations:
        createdAt: 'FIXME'
    spec:
      containers:
        - image: alpine:3.10.2
          imagePullPolicy: IfNotPresent
          name: test
          args:
            - ash
            - -c
            - |
              trap exit TERM
              while :; do
                sleep 1
              done
          readinessProbe:
            exec:
              command:
                - "true"
            initialDelaySeconds: 5
          resources:
            requests:
              cpu: 100m
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: test
  namespace: default
spec:
  minReplicas: 2
  maxReplicas: 6
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: test
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 100

# We replace "FIXME" by current time so that we can apply it many times.
$ sed -e "s/FIXME/$(date +%s)/" test.yaml | kubectl apply -f -

Step2

Open 2 new terminals, and monitor the test pod and test horizontal pod autoscaler.

$ while :; do date; kubectl get pod --sort-by=.metadata.creationTimestamp | grep -E '^test-'; sleep 2; done
Tue Sep 17 11:19:25 JST 2019
test-847887944b-g7kh7                                  1/1     Running             0          16s
test-847887944b-mphsz                                  0/1     ContainerCreating   0          1s
Tue Sep 17 11:19:27 JST 2019
test-847887944b-g7kh7                                  1/1     Running     0          18s
test-847887944b-mphsz                                  0/1     Running     0          3s
...

$ kubectl get hpa test --watch
NAME   REFERENCE         TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
test   Deployment/test   <unknown>/100%   2         6         0          0s
test   Deployment/test   <unknown>/100%   2         6         1          15s
test   Deployment/test   <unknown>/100%   2         6         2          30s
...

Step3

Wait until the test horizontal pod autoscaler is stable, that is, changing TARGETS from <unknown> to a certain number (eg: 1%).

$ kubectl get hpa test --watch
NAME   REFERENCE         TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
test   Deployment/test   <unknown>/100%   2         6         0          0s
test   Deployment/test   <unknown>/100%   2         6         1          15s
...
test   Deployment/test   1%/100%          2         6         2          3m30s

Step4

Re-apply test.yaml so that new deployment will be created.

$ sed -e "s/FIXME/$(date +%s)/" test.yaml | kubectl apply -f -

Step5

Then you will see the pods are increasing from 2 to 3.

$ while :; do date; kubectl get pod --sort-by=.metadata.creationTimestamp | grep -E '^test-'; sleep 2; done
...
Tue Sep 17 11:20:57 JST 2019
test-847887944b-g7kh7                                  1/1     Running     0          108s
test-59b4bccf57-mfbk8                                  1/1     Running     0          19s
test-59b4bccf57-wsfkc                                  0/1     Running     0          6s
test-59b4bccf57-9zqrz                                  0/1     Running     0          3s