tormath1/README.md

## README.md

      
    Raw
  

              README.md
            
          
    This is a demo of updating Kubernetes version on Flatcar without updating the OS leveraging Systemd sysext and Kured.
notes:

It has been tested on a simple control-plane node on Qemu but can be easily extended to CAPI.
The kubernetes sysext image is provided by https://github.com/flatcar/sysext-bakery.git

Boot an instance with the following Butane config.yaml:
variant: flatcar
version: 1.0.0
systemd:
  units:
    - name: systemd-sysupdate@kubernetes.service
    - name: bootstrap-sysupdate@kubernetes.service
      enabled: true
    - name: systemd-sysupdate@kubernetes.timer
      enabled: true
    - name: bootstrap-sysupdate@.service
      contents: |
        [Unit]
        Description=Bootstrap the %I component
        ConditionPathIsSymbolicLink=!/etc/extensions/%I.raw
        ConditionVirtualization=!container
        DefaultDependencies=no

        Before=systemd-sysext.service

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/lib/systemd/systemd-sysupdate --component=%I update

        [Install]
        WantedBy=sysinit.target
    - name: systemd-sysupdate@.timer
      contents: |
        [Unit]
        Description=Automatic System Update
        Documentation=man:systemd-sysupdate.service(8)
        
        # For containers we assume that the manager will handle updates. And we likely
        # can't even access our backing block device anyway.
        ConditionVirtualization=!container
        
        [Timer]
        # Trigger the update 15min after boot, and then – on average – every 6h, but
        # randomly distributed in a 2h…6h interval. In addition trigger things
        # persistently once on each Saturday, to ensure that even on systems that are
        # never booted up for long we have a chance to to do the update.
        OnBootSec=15min
        OnUnitActiveSec=2h
        OnCalendar=Sat
        RandomizedDelaySec=4h
        Persistent=yes
        
        [Install]
        WantedBy=timers.target
    - name: systemd-sysupdate@.service
      contents: |
        [Unit]
        Description=Automatic System Update
        Documentation=man:systemd-sysupdate.service(8)
        Wants=network-online.target
        After=network-online.target
        ConditionVirtualization=!container
        
        [Service]
        Type=simple
        NotifyAccess=main
        ExecStart=/usr/lib/systemd/systemd-sysupdate --component=%I update
        ExecStartPost=/usr/bin/touch /run/reboot-required
        CapabilityBoundingSet=CAP_CHOWN CAP_FOWNER CAP_FSETID CAP_MKNOD CAP_SETFCAP CAP_SYS_ADMIN CAP_SETPCAP CAP_DAC_OVERRIDE CAP_LINUX_IMMUTABLE
        NoNewPrivileges=yes
        MemoryDenyWriteExecute=yes
        ProtectHostname=yes
        RestrictRealtime=yes
        RestrictNamespaces=net
        RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
        SystemCallFilter=@system-service @mount
        SystemCallErrorNumber=EPERM
        SystemCallArchitectures=native
        LockPersonality=yes
        
        [Install]
        Also=systemd-sysupdate.timer
storage:
  files:
    - path: /etc/sysupdate.kubernetes.d/kubernetes.conf
      contents:
        inline: |
          [Transfer]
          Verify=false
          
          [Source]
          Type=url-file
          Path=http://192.168.1.17:8000/
          MatchPattern=kubernetes-@v.raw
          
          [Target]
          InstancesMax=3
          Type=regular-file
          Path=/opt/extensions/kubernetes
          CurrentSymlink=/etc/extensions/kubernetes.raw
note: Don't forget to update the IP/URL of the systemd-sysext source. For CAPI, I use GCS Bucket for example.
SSH on the instance and assert everything works fine and start the Kubernetes control plane:
$ systemd-sysext status
HIERARCHY EXTENSIONS SINCE
/opt      none       -
/usr      kubernetes Fri 2023-07-28 08:55:01 UTC
          oem-qemu
$ ls -l /etc/extensions/
total 0
lrwxrwxrwx. 1 root root 37 Jul 28 08:54 kubernetes.raw -> /opt/extensions/kubernetes-1.26.6.raw
lrwxrwxrwx. 1 root root 32 Jul 28 08:54 oem-qemu.raw -> /oem/sysext/oem-qemu-initial.raw
$ sudo kubeadm init
...
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ kubectl apply -f https://docs.projectcalico.org/archive/v3.23/manifests/calico.yaml

Install Kured to manage the reboot mechanism when there is a new update:
$ latest=$(curl -s https://api.github.com/repos/kubereboot/kured/releases | jq -r '.[0].tag_name')
$ curl -O -fsSL "https://github.com/kubereboot/kured/releases/download/$latest/kured-$latest-dockerhub.yaml"
# edit kured-$latest-dockerhub.yaml to have a --period to 10s for demo purposes
$ kubectl apply -f kured-$latest-dockerhub.yaml

Force the update for demo purposes (otherwise it's a regular systemd timer). The node is running version 1.26.6 before the update:
$ kubectl get nodes
NAME        STATUS   ROLES           AGE     VERSION
localhost   Ready    control-plane   6m57s   v1.26.6
$ sudo systemctl start systemd-sysupdate
$ journalctl -f -u systemd-sysupdate
...
Jul 28 09:05:20 localhost systemd-sysupdate[5895]: Updated symlink '/etc/extensions/kubernetes.raw' → '../../opt/extensions/kubernetes-1.27.3.raw'.
Broadcast message from root@localhost (Fri 2023-07-28 09:20:12 UTC):

The system will reboot now!

Connection to 127.0.0.1 closed by remote host.
Connection to 127.0.0.1 closed.

You can SSH back on the node and assert that Kubernetes has been upgraded:
$ kubectl get nodes
NAME        STATUS   ROLES           AGE   VERSION
localhost   Ready    control-plane   13m   v1.27.3