In this entry I will address the subject of the autoscale of the Stateful Set Issues source, at this moment Kubernetes does not support to auto scale of the Stateful Set, to solve this need, I decided to program two slug-autoscaler tools to monitor the replicas to modify them according to the state of the CPU and slug-labalancer to customize the load balance. I also want to include support for properly scaling crunchy-containers.
Image: tedezed/slug-statefulset-autoscaler:2.0 or latest
Started by copy the repository:
git clone https://github.com/Tedezed/slug-containers.git
cd slug-containers/statefulset_autoscaler/kube/
Create the slug-autoscaler-rc:
kubectl create -f slug-autoscaler-rc.yaml
Check the status:
kubectl get rc,pod --all-namespaces | grep slug
kube-system rc/slug-autoscaler 1 1 1 1m
kube-system po/slug-autoscaler-0lvk2 1/1 Running 0 1
- autoscaler: For add list to autoscale stateful set.
- autoscaler_percent_cpu: Autoscaling if percent CPU is greater than number.
- autoscaler_count: Number of query for the next attempt to autoscale.
- autoreduce_normal: Autoscaling for the simple HA (Regardless of masters or slaves).
- autoreduce_percent_cpu: Reduce scale if percent CPU if below than number.
- min_replicas
- max_replicas
You need to add limits of resources to calculate the CPU percentage:
resources:
limits:
cpu: 150m
requests:
cpu: 75m
Create the set-example-nginx
kubectl create -f set-example-nginx.yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: nginx
labels:
app: nginx
spec:
serviceName: "nginx"
replicas: 5
template:
metadata:
labels:
app: nginx
annotations:
pod.alpha.kubernetes.io/initialized: "true"
slug-autoscaler/autoscaler: "true"
slug-autoscaler/autoscaler_percent_cpu: "50"
slug-autoscaler/autoscaler_count: "0"
slug-autoscaler/autoreduce_normal: "true"
slug-autoscaler/autoreduce_percent_cpu: "10"
slug-autoscaler/min_replicas: "3"
slug-autoscaler/max_replicas: "8"
slug-autoscaler/sts_owner_name: "nginx"
spec:
containers:
- name: nginx
image: nginx
env:
- name: POD_NS
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
ports:
- name: client
containerPort: 80
resources:
limits:
cpu: 150m
memory: 300Mi
requests:
cpu: 75m
memory: 150Mi
You can balance the service with services with kubernetes service or use the Slug dynamic balancer to customize the balance, this last one believes it to solve a series of problems with the balance of services in kubernetes, but generally it is not necessary to use it. In this case we will use the balance of services of kubernetes.
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx
name: nginx
spec:
ports:
- name: http
port: 80
protocol: TCP
selector:
app: nginx
kubectl logs slug-autoscaler-0lvk2 -n kube-system
As the CPU percentage is below autoreduce_percent_cpu, slug-autoscaler reduces the number of replicas to the number min_replicas.
Start Slug StatefulSet Autoscaler v.0.0.1
[StatefulSet] nginx 0 CPU - Autoscale in 50 CPU reduce in 10 CPU
[AUTOSCALING] nginx replicas: min 1 max 8
[AUTOSCALING] nginx replicas: 5 to 4
[INFO] StatefulSet autoscaler 1
Seelp 5s for next query
[StatefulSet] nginx 0 CPU - Autoscale in 50 CPU reduce in 10 CPU
[INFO] Seelp StatefulSet nginx replicas 4, attempts 2
[INFO] StatefulSet autoscaler 0
Seelp 5s for next query
[StatefulSet] nginx 0 CPU - Autoscale in 50 CPU reduce in 10 CPU
[INFO] Seelp StatefulSet nginx replicas 4, attempts 1
[INFO] StatefulSet autoscaler 0
Seelp 5s for next query
[StatefulSet] nginx 0 CPU - Autoscale in 50 CPU reduce in 10 CPU
[AUTOSCALING] nginx replicas: min 1 max 8
[AUTOSCALING] nginx replicas: 4 to 3
[INFO] StatefulSet autoscaler 1
Seelp 5s for next query
[StatefulSet] nginx 0 CPU - Autoscale in 50 CPU reduce in 10 CPU
[INFO] Seelp StatefulSet nginx replicas 3, attempts 1
[INFO] StatefulSet autoscaler 0
Seelp 5s for next query
I will use the Apache HTTP server benchmarking tool to run an application test
ab -n 900000 -c 2000 your_ginx
[StatefulSet] nginx 80 CPU - Autoscale in 50 CPU reduce in 10 CPU
[AUTOSCALING] nginx replicas: min 1 max 8
[AUTOSCALING] nginx replicas: 5 to 6
[INFO] StatefulSet autoscaler 1
Seelp 5s for next query
[StatefulSet] nginx 80 CPU - Autoscale in 50 CPU reduce in 10 CPU
[INFO] Seelp StatefulSet nginx replicas 6, attempts 2
[INFO] StatefulSet autoscaler 0
Seelp 5s for next query
[StatefulSet] nginx 80 CPU - Autoscale in 50 CPU reduce in 10 CPU
[INFO] Seelp StatefulSet nginx replicas 6, attempts 1
[INFO] StatefulSet autoscaler 0
Seelp 5s for next query
[StatefulSet] nginx 80 CPU - Autoscale in 50 CPU reduce in 10 CPU
[INFO] Seelp StatefulSet nginx replicas 6, attempts 0
[INFO] StatefulSet autoscaler 0
Seelp 5s for next query
[StatefulSet] nginx 80 CPU - Autoscale in 50 CPU reduce in 10 CPU
[AUTOSCALING] nginx replicas: min 1 max 8
[AUTOSCALING] nginx replicas: 6 to 7
[INFO] StatefulSet autoscaler 1
Seelp 5s for next query
Image: tedezed/slug-statefulset-autoscaler:0.1
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: nginx
labels:
autoscaler: "true"
autoscaler_percent_cpu: "50"
autoscaler_count: "0"
autoreduce_normal: "true"
autoreduce_percent_cpu: "10"
min_replicas: "1"
max_replicas: "8"
app: nginx
spec:
serviceName: "nginx"
replicas: 5
template:
metadata:
labels:
app: nginx
annotations:
pod.alpha.kubernetes.io/initialized: "true"
spec:
containers:
- name: nginx
image: nginx
env:
- name: POD_NS
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
ports:
- name: client
containerPort: 80
resources:
limits:
cpu: 150m
requests:
cpu: 75m