Skip to content

Instantly share code, notes, and snippets.

@moondev

moondev/upgrade.md

Last active September 25, 2019 23:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save moondev/8c094f6d1bd9ca475a249d1b14b35874 to your computer and use it in GitHub Desktop.
Save moondev/8c094f6d1bd9ca475a249d1b14b35874 to your computer and use it in GitHub Desktop.
Download ZIP
K8S upgrade 1.10.x -> 1.11.x -> 1.12.x -> 1.13.x -> 1.14.x
Raw
upgrade.md

1.10.x -> 1.14.x

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md https://v1-10.docs.kubernetes.io/docs/setup/independent/high-availability/

example etcdctl.sh helper script on etcd-1

#!/bin/bash

export ETCDCTL_API=3

ARGS="$@"

#1580e4b729cdab7b, started, ip-172-31-83-183, https://172.31.83.183:2380, https://172.31.83.183:2379
#1cd223d46fd0c0bf, started, ip-172-31-86-252, https://172.31.86.252:2380, https://172.31.86.252:2379
#d9517cfa32ed7a98, started, ip-172-31-92-169, https://172.31.92.169:2380, https://172.31.92.169:2379

etcdctl --endpoints=https://10.10.129.205:2379,https://10.10.128.112:2379,https://10.10.123.13:2379 \
        --cacert=/etc/kubernetes/pki/etcd/ca.pem \
        --cert=/etc/kubernetes/pki/etcd/peer.pem \
        --key=/etc/kubernetes/pki/etcd/peer-key.pem $ARGS

Upgrade each master node control plane static pods

Each control plane node should have a kubeadm.yaml on the filesystem. This should look like the following, where 10.10.65.247 is the haproxy kube-apiserver loadbalancer, and 10.10.65.254 is the private ip of the current node.

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
api:
  advertiseAddress: 10.10.123.247
  controlPlaneEndpoint: 10.10.123.247
etcd:
  endpoints:
  - https://10.10.129.205:2379
  - https://10.10.128.112:2379
  - https://10.10.123.13:2379
  caFile: /etc/kubernetes/pki/etcd/ca.pem
  certFile: /etc/kubernetes/pki/etcd/client.pem
  keyFile: /etc/kubernetes/pki/etcd/client-key.pem
networking:
  podSubnet: 192.168.0.0/16
apiServerCertSANs:
- 10.10.123.247
- 10.10.90.233
apiServerExtraArgs:
  endpoint-reconciler-type: lease

Download kubeadm v1.11.10 directly on top of the current kubeadm

curl -sSL https://dl.k8s.io/release/v1.11.10/bin/linux/amd64/kubeadm > /usr/bin/kubeadm
chmod a+rx /usr/bin/kubeadm
kubeadm version -o short

Plan the upgrade (dry-run)

kubeadm --config kubeadm.yaml upgrade plan v1.11.10 --allow-experimental-upgrades

Apply the upgrade

kubeadm --config kubeadm.yaml upgrade apply v1.11.10 --allow-experimental-upgrades

Perform the same steps above on the other master nodes. Only when this is complete should the kubelets be upgraded.

Check the images

kubectl get pods -n kube-system -l tier=control-plane -o yaml | grep 'image:'

Upgrade each master node kubeadm, kubelet and kubectl packages

Drain the master

kubectl drain `hostname` --ignore-daemonsets

Upgrade kubeadm on the masters

apt-get upgrade kubelet=1.11.10-00 kubeadm=1.11.10-00 kubectl=1.11.10-00 kubernetes-cni=0.6.0-00

systemctl daemon-reload
systemctl restart kubelet
kubectl uncordon `hostname`

Check nodes and container images

kubectl get nodes -o wide
kubectl get pods --all-namespaces -o yaml | grep 'image:'

Upgrade each worker

kubectl drain `k8s-w-1` --ignore-daemonsets
apt-get upgrade kubelet=1.11.10-00 kubeadm=1.11.10-00 kubectl=1.11.10-00 kubernetes-cni=0.6.0-00
kubeadm upgrade node config --kubelet-version v1.11.10

Copy the /var/lib/kubelet/kubeadm-flags.env from master to /var/lib/kubelet/kubeadm-flags.env on the worker

systemctl daemon-reload
systemctl restart kubelet
kubectl uncordon k8s-w-1

Check nodes and workers

kubectl get nodes -o wide
kubectl get pods --all-namespaces -o yaml | grep 'image:'

1.11.10 -> 1.12.8

Upgrade packages

apt-get upgrade kubeadm=1.12.8-00

Download config on EVERY master

kubectl get configmap -n kube-system kubeadm-config -o jsonpath={.data.MasterConfiguration} > kubeadm-config.yaml

Example:

[root@rhel-master-1 ~]# cat kubeadm-config.yaml
api:
  advertiseAddress: 10.10.65.247
  bindPort: 6443
  controlPlaneEndpoint: 10.10.65.247
apiServerCertSANs:
- 10.10.65.247
- 10.10.66.0
apiServerExtraArgs:
  authorization-mode: Node,RBAC
  endpoint-reconciler-type: lease
apiVersion: kubeadm.k8s.io/v1alpha2
auditPolicy:
  logDir: /var/log/kubernetes/audit
  logMaxAge: 2
  path: ""
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
etcd:
  external:
    caFile: /etcd/ca.crt
    certFile: /etcd/peer.crt
    endpoints:
    - https://10.10.65.248:2379
    keyFile: /etcd/peer.key
imageRepository: k8s.gcr.io
kind: MasterConfiguration
kubeProxy:
  config:
    bindAddress: 0.0.0.0
    clientConnection:
      acceptContentTypes: ""
      burst: 10
      contentType: application/vnd.kubernetes.protobuf
      kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
      qps: 5
    clusterCIDR: 172.22.0.0/17
    configSyncPeriod: 15m0s
    conntrack:
      max: null
      maxPerCore: 32768
      min: 131072
      tcpCloseWaitTimeout: 1h0m0s
      tcpEstablishedTimeout: 24h0m0s
    enableProfiling: false
    healthzBindAddress: 0.0.0.0:10256
    hostnameOverride: ""
    iptables:
      masqueradeAll: false
      masqueradeBit: 14
      minSyncPeriod: 0s
      syncPeriod: 30s
    ipvs:
      excludeCIDRs: null
      minSyncPeriod: 0s
      scheduler: ""
      syncPeriod: 30s
    metricsBindAddress: 127.0.0.1:10249
    mode: ""
    nodePortAddresses: null
    oomScoreAdj: -999
    portRange: ""
    resourceContainer: /kube-proxy
    udpIdleTimeout: 250ms
kubeletConfiguration:
  baseConfig:
    address: 0.0.0.0
    authentication:
      anonymous:
        enabled: false
      webhook:
        cacheTTL: 2m0s
        enabled: true
      x509:
        clientCAFile: /etc/kubernetes/pki/ca.crt
    authorization:
      mode: Webhook
      webhook:
        cacheAuthorizedTTL: 5m0s
        cacheUnauthorizedTTL: 30s
    cgroupDriver: cgroupfs
    cgroupsPerQOS: true
    clusterDNS:
    - 172.22.128.10
    clusterDomain: cluster.local
    containerLogMaxFiles: 5
    containerLogMaxSize: 10Mi
    contentType: application/vnd.kubernetes.protobuf
    cpuCFSQuota: true
    cpuManagerPolicy: none
    cpuManagerReconcilePeriod: 10s
    enableControllerAttachDetach: true
    enableDebuggingHandlers: true
    enforceNodeAllocatable:
    - pods
    eventBurst: 10
    eventRecordQPS: 5
    evictionHard:
      imagefs.available: 15%
      memory.available: 100Mi
      nodefs.available: 10%
      nodefs.inodesFree: 5%
    evictionPressureTransitionPeriod: 5m0s
    failSwapOn: true
    fileCheckFrequency: 20s
    hairpinMode: promiscuous-bridge
    healthzBindAddress: 127.0.0.1
    healthzPort: 10248
    httpCheckFrequency: 20s
    imageGCHighThresholdPercent: 85
    imageGCLowThresholdPercent: 80
    imageMinimumGCAge: 2m0s
    iptablesDropBit: 15
    iptablesMasqueradeBit: 14
    kubeAPIBurst: 10
    kubeAPIQPS: 5
    makeIPTablesUtilChains: true
    maxOpenFiles: 1000000
    maxPods: 110
    nodeStatusUpdateFrequency: 10s
    oomScoreAdj: -999
    podPidsLimit: -1
    port: 10250
    registryBurst: 10
    registryPullQPS: 5
    resolvConf: /etc/resolv.conf
    rotateCertificates: true
    runtimeRequestTimeout: 2m0s
    serializeImagePulls: true
    staticPodPath: /etc/kubernetes/manifests
    streamingConnectionIdleTimeout: 4h0m0s
    syncFrequency: 1m0s
    volumeStatsAggPeriod: 1m0s
kubernetesVersion: v1.11.6
networking:
  dnsDomain: cluster.local
  podSubnet: 172.22.0.0/17
  serviceSubnet: 172.22.128.0/17
nodeRegistration: {}
unifiedControlPlaneImage: ""

api.advertiseAddress to the local node’s IP address

kubeadm upgrade plan v1.12.8 --config kubeadm-config.yaml
kubeadm upgrade apply v1.12.8 --config kubeadm-config.yaml

Update only kubelet and kubectl

kubectl drain `hostname` --ignore-daemonsets
apt-get upgrade kubelet=1.12.8-00 kubectl=1.12.8-00
systemctl daemon-reload
systemctl restart kubelet
kubectl uncordon `uncordon`

Check nodes and images

kubectl get nodes -o wide
kubectl get pods --all-namespaces -o yaml | grep 'image:'

upgrade worker

kubectl drain `hostname` --ignore-daemonsets
apt-get upgrade kubelet=1.12.8-00 kubeadm=1.12.8-00 kubectl=1.12.8-00
kubeadm upgrade node config --kubelet-version v1.12.5
systemctl daemon-reload
systemctl restart kubelet

one-liner option

kubectl drain `hostname` --ignore-daemonsets; yum upgrade -y kubelet-1.12.5-0 kubeadm-1.12.5-0 kubectl-1.12.5-0 --disableexcludes=kubernetes; kubeadm upgrade node config --kubelet-version v1.12.5; systemctl daemon-reload; systemctl restart kubelet; kubectl uncordon `hostname`

Upgrade etcd to 3.2.24 in /etc/kubernetes/manifests/etcd.yaml

##Upgrade CNI

check calico version

kubectl get pods -n kube-system -l=k8s-app=calico-node -o yaml | grep 'image: '

Download updated calico manifests

wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml

wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

edit the cluster cidr to 172.22.0.0/17 in calico manifest

apply manifests

Upgrade docker on master node

drain master

kubectl drain `hostname` --ignore-daemonsets
systemctl stop kubelet
systemctl stop docker

apt-get install docker-ce=18.06.2~ce~3-0~ubuntu

systemctl start docker
docker version
docker info | grep Cgroup
systemctl start kubelet
kubectl uncordon `hostname`
kubectl get nodes -o wide
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment