Skip to content

Instantly share code, notes, and snippets.

@rosskirkpat
Created October 13, 2022 01:31
Show Gist options
  • Save rosskirkpat/57aa392a4b44cca3d48dfe58b5716954 to your computer and use it in GitHub Desktop.
Save rosskirkpat/57aa392a4b44cca3d48dfe58b5716954 to your computer and use it in GitHub Desktop.
Configure max pods for RKE, RKE2, and k3s clusters

How to configure max pods for the SUSE Rancher Kubernetes distributions

Considerations when Increasing the Max Pod Count

Changing the max-pods on an active cluster with workloads is generally a safe procedure when target number of max-pods is <=250. When the goal number of max-pods is >250, the additional considerations mentioned above require a deletion of all currently running pods.

If increasing max-pods to >250, there are additional considerations and changes required. The in-cluster IP management configuration needs to be modified as the default is a /16 split into one /24 for each node in the cluster. This comes to a limit of about 256 nodes with roughly 253 pods per node.

--max-pods int32     Default: 110
    Number of Pods that can run on this Kubelet. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)

kubelet config file

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 250

RKE1

Modify the cluster.yaml

Example RKE Full cluster.yml

rancher_kubernetes_engine_config:
    ...
    services:
        kubelet:
            extra_args:
                max-pods: 250

RKE2

RKE2 Configuration File

RKE2 Server

RKE2 Server Configuration Options

curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="server" sh -

cat << EOF >> /etc/rancher/rke2/kubelet-server.config
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 250
EOF

cat << EOF >> /etc/rancher/rke2/config-server.yaml
write-kubeconfig-mode: "0644"
kubelet-arg: "/etc/rancher/rke2/kubelet-server.config"
EOF

rke2 server --config=/etc/rancher/rke2/config-server.yaml --kubelet-arg=config=/etc/rancher/rke2/kubelet.config

RKE2 Agent

RKE2 Agent Configuration Options

curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" sh -

cat << EOF >> /etc/rancher/rke2/kubelet-agent.config
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 250
EOF

cat << EOF >> /etc/rancher/rke2/config-agent.yaml
write-kubeconfig-mode: "0644"
kubelet-arg: "/etc/rancher/rke2/kubelet-agent.config"
EOF

rke2 agent --config=/etc/rancher/rke2/config-agent.yaml --kubelet-arg=config=/etc/rancher/rke2/kubelet.config

K3S

K3S Configuration File

k3s server

K3S Server Configuration Options

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --kubelet-arg=config=/etc/rancher/k3s/kubelet-server.config" sh -

cat << EOF >> /etc/rancher/k3s/kubelet-server.config
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 250
EOF

cat << EOF >> /etc/rancher/k3s/config-server.yaml
write-kubeconfig-mode: "0644"
kubelet-arg: "/etc/rancher/k3s/kubelet-server.config"
EOF

k3s server --config=/etc/rancher/k3s/config-server.yaml --kubelet-arg=config=/etc/rancher/k3s/kubelet-server.config

k3s agent

K3S Agent Configuration Options

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent --kubelet-arg=config=/etc/rancher/k3s/kubelet-server.config" sh -

cat << EOF >> /etc/rancher/k3s/kubelet-agent.config
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 250
EOF


cat << EOF >> /etc/rancher/k3s/config-agent.yaml
write-kubeconfig-mode: "0644"
kubelet-arg: "/etc/rancher/k3s/kubelet-agent.config"
EOF

k3s agent --config=/etc/rancher/k3s/config-agent.yaml --kubelet-arg=config=/etc/rancher/k3s/kubelet-agent.config

Note You also have to reconfigure the k3s systemd service /etc/systemd/system/k3s.service

ExecStart=/usr/local/bin/k3s \
    server \
        '--disable' \
        'servicelb' \
        '--disable' \
        'traefik' \
        '--kubelet-arg=config=/etc/rancher/k3s/kubelet-server.config'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment