Skip to content

Instantly share code, notes, and snippets.

@gbevan

gbevan/README.md

Last active July 20, 2023 08:56
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save gbevan/7fe51e4110ed588fc346e4f9a267df1e to your computer and use it in GitHub Desktop.
Save gbevan/7fe51e4110ed588fc346e4f9a267df1e to your computer and use it in GitHub Desktop.
Download ZIP
PoC - Kairos Build ISO from an Installed Linux VM as a base
Raw
README.md

Notes on PoC for Building a Kairos ISO Image from an Installed Linux VM as a Base

Create the base VM

Boot from Rocky 9 ISO
Minimal image + standard + guest agents

On the booted img

dnf update -y
dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf install -y wget docker-ce docker-ce-cli containerd.io
systemctl start docker
systemctl enable docker

fallocate --length 16GiB /swapfile
mkswap /swapfile
swapon /swapfile

setenforce 0
and edit /etc/selinux/config

systemctl enable getty@tty1.service
systemctl enable getty@tty2.service
systemctl enable getty@tty3.service

ensure sudoers and default user set

Create Docker Image from the Installed VM

tar cvf - \
        --numeric-owner \
        --exclude=/proc --exclude=/tmp --exclude=/var/tmp --exclude=/mnt --exclude=/sys --exclude=/dev --exclude=/run \
        --exclude=/swapfile \
        --exclude='/var/lib/docker/overlay2/*' \
        --exclude='/root/*' \
        / | docker import - myimg

Create a Dockerfile for the Kairos Framework + any extras

ARG BASE_IMAGE=myimg

FROM $BASE_IMAGE as base

RUN echo "install_weak_deps=False" >> /etc/dnf/dnf.conf
RUN mkdir -m 1777 /tmp
RUN mkdir -m 1777 /var/tmp
RUN mkdir -m 755 /run
RUN dnf install -y epel-release && dnf clean all
RUN dnf update -y
RUN dnf makecache
RUN dnf upgrade -y
# Cleanout build base
RUN dnf remove -y \
	docker-ce \
	docker-ce-cli \
	docker-compose-plugin \
	docker-ce-rootless-extras
RUN dnf install -y \
    audit \
    device-mapper \
    dosfstools \
    dracut \
    dracut-live \
    dracut-network \
    dracut-squash \
    e2fsprogs \
    efibootmgr \
    epel-release \
    gawk \
    grub2 \
    grub2-efi-x64 \
    grub2-efi-x64-modules \
    grub2-pc \
    kernel \
    kernel-modules \
    kernel-modules-extra \
    livecd-tools \
    lvm2 \
    nano \
    openssh-server \
    parted \
    polkit \
    qemu-guest-agent \
    rsync \
    shim-x64 \
    squashfs-tools \
    sudo \
    systemd \
    systemd-networkd \
    systemd-resolved \
    systemd-timesyncd \
    tar \
    which \
    https://zfsonlinux.org/epel/zfs-release-2-2.el9.noarch.rpm \
    curl \
    podman \
    && dnf clean all
    #cloud-init 

RUN mkdir -p /run/lock
RUN touch /usr/libexec/.keep
RUN systemctl enable getty@tty1.service || /bin/true
RUN systemctl enable getty@tty2.service || /bin/true
RUN systemctl enable getty@tty3.service || /bin/true
RUN systemctl enable systemd-networkd || /bin/true
RUN systemctl enable systemd-resolved || /bin/true
RUN systemctl disable dnf-makecache.service || /bin/true
RUN systemctl enable sshd || /bin/true

# Copy the Kairos framework files. We use master builds here for fedora. See https://quay.io/repository/kairos/framework?tab=tags for a list
#COPY --from=quay.io/kairos/framework:v2.2.0_rockylinux / /
COPY --from=quay.io/kairos/framework:v2.3.0_rockylinux / /

# Set the Kairos arguments in os-release file to identify your Kairos image
FROM quay.io/kairos/osbuilder-tools:latest as osbuilder
RUN zypper install -y gettext
RUN mkdir /workspace
COPY --from=base /etc/os-release /workspace/os-release
# You should change the following values according to your own versioning and other details
RUN OS_NAME=kairos-my-rockylinux \
  OS_VERSION=v9 \
  OS_ID="kairos" \
  BUG_REPORT_URL="https://github.com/YOUR_ORG/YOUR_PROJECT/issues" \
  HOME_URL="https://github.com/YOUR_ORG/YOUR_PROJECT" \
  OS_REPO="quay.io/YOUR_ORG/core-rockylinux" \
  OS_LABEL="latest" \
  GITHUB_REPO="YOUR_ORG/YOUR_PROJECT" \
  VARIANT="core" \
  FLAVOR="rhel" \
  /update-os-release.sh

FROM base
COPY --from=osbuilder /workspace/os-release /etc/os-release

# Activate Kairos services
RUN systemctl enable cos-setup-reconcile.timer && \
          systemctl enable cos-setup-fs.service && \
          systemctl enable cos-setup-boot.service && \
          systemctl enable cos-setup-network.service

# Increase size of active.img etc so kairos-agent install doesnt (silently) fail with
# No space left on device (and dumps stdout file copy details slowly to console...)
RUN \
	sed -i -e 's/size:.*/size: 4000/' /etc/elemental/config.yaml

# Prevent kairos-agent install systemd unit from slow dumping to /dev/tty1 console
# TODO: revert to normal after fixing active.img disk sizes etc
#RUN \
#	sed -i -e '/tty/d' /etc/systemd/system/kairos.service

# Install k3s airgap
RUN \
	mkdir -p /var/lib/rancher/k3s/agent/images/ && \
	cd /var/lib/rancher/k3s/agent/images/ && \
	wget https://github.com/k3s-io/k3s/releases/download/v1.25.11%2Bk3s1/k3s-airgap-images-amd64.tar && \
	ls -la && \
	cd /usr/bin && \
	wget https://github.com/k3s-io/k3s/releases/download/v1.25.11%2Bk3s1/k3s && \
	chmod 755 k3s && \
	wget https://get.k3s.io -O k3s-install.sh && \
	chmod 700 k3s-install.sh && \
	ls -la

ENV INSTALL_K3S_BIN_DIR="/usr/bin"
RUN \
	set -o pipefail; \
	INSTALL_K3S_SKIP_DOWNLOAD=true \
	INSTALL_K3S_SELINUX_WARN=true \
	INSTALL_K3S_SKIP_START="true" \
	INSTALL_K3S_SKIP_ENABLE="true" \
	INSTALL_K3S_SKIP_SELINUX_RPM="true" \
	k3s-install.sh 2>&1 | tee -a /root/k3s-install.log || cat /root/k3s-install.log && \
	rm -f /usr/bin/k3s-install.sh && \
	systemctl enable k3s.service
#	INSTALL_K3S_SELINUX_WARN=true \
#	INSTALL_K3S_SKIP_START="true" \
#	INSTALL_K3S_SKIP_ENABLE="true" \
#	INSTALL_K3S_SKIP_SELINUX_RPM="true" \
#	k3s-install.sh agent && \

# Install arkade
RUN \
	OWNER=alexellis; \
	REPO=arkade; \
	version=$(curl -sI https://github.com/$OWNER/$REPO/releases/latest | grep -i "location:" | awk -F"/" '{ printf "%s", $NF }' | tr -d '\r') && \
	wget https://github.com/$OWNER/$REPO/releases/download/$version/arkade -O /usr/bin/arkade && \
	chmod 755 /usr/bin/arkade && \
	ln -s /usr/bin/arkade /usr/bin/ark

# Install helm
RUN \
	cd /tmp && \
	wget https://get.helm.sh/helm-v3.12.2-linux-amd64.tar.gz -O helm.tar.gz && \
	tar zxvf helm.tar.gz && \
	mv linux-amd64/helm /usr/bin

## Generate initrd
RUN kernel=$(ls /boot/vmlinuz-* | head -n1) && \
            ln -sf "${kernel#/boot/}" /boot/vmlinuz
RUN kernel=$(ls /lib/modules | head -n1) && \
            dracut -v -N -f "/boot/initrd-${kernel}" "${kernel}" && \
            ln -sf "initrd-${kernel}" /boot/initrd && depmod -a "${kernel}"
RUN rm -rf /boot/initramfs-*

RUN sed -i -e '/\/home/d' -e '/swap/d' -e '/\/boot/d' /etc/fstab

Build the Kairos Layered Docker Image

docker build -t myimg2 .

Create the Kairos ISO Image

Notes take from https://kairos.io/docs/installation/automated/#iso-remastering

(assumimg cd ~root)

IMAGE=myimg2

mkdir -p files-iso/boot/grub2

# You can replace this step with your own grub config. This GRUB configuration is the boot menu of the ISO
wget https://raw.githubusercontent.com/kairos-io/kairos/master/overlay/files-iso/boot/grub2/grub.cfg -O files-iso/boot/grub2/grub.cfg

# Copy the config file
cp -rfv cloud_init.yaml files-iso/cloud_config.yaml

# Pull the image locally
NONEED, ALREADY HERE FROM ABOVE $ docker pull $IMAGE

# Optionally, modify the image here!
docker run --entrypoint /bin/bash --name changes -ti $IMAGE
        commented out /boot, home and swap from /etc/fstab - now in Dockerfile
docker commit changes $IMAGE

# Build an ISO with $IMAGE
docker -D run \
        -v $PWD:/cOS \
        -v /var/run/docker.sock:/var/run/docker.sock \
        -i --rm \
        --name custom-iso \
        quay.io/kairos/osbuilder-tools:latest \
        build-iso --debug --date=false --local --overlay-iso /cOS/files-iso $IMAGE --output /cOS/

Upgrading a Deployed Kairos Instance from an Updated Docker Image

In my test, I made a change to the above Dockerfile and rebuilt the image. From that image I deployed a container which I then immediately terminated and left in a stopped state. The container was called "changes":

docker export changes > mynewimg2.tar

scp mynewimg2.tar me@mykairosvm:
``
then on the kairos vm:

mkdir d cd d tar xvf mynewimg2.tar

and upgrade the kairos instance with the new image:

kairos-agent upgrade --source dir:d/

INFO[2023-07-19T14:43:56+01:00] kairos-agent version v2.1.3
INFO[2023-07-19T14:43:56+01:00] Applying 'before-upgrade' hook
INFO[2023-07-19T14:43:56+01:00] Running before-upgrade hook
INFO[2023-07-19T14:43:56+01:00] deploying image d/ to /run/initramfs/cos-state/cOS/transition.img
INFO[2023-07-19T14:43:56+01:00] Creating file system image /run/initramfs/cos-state/cOS/transition.img
INFO[2023-07-19T14:43:56+01:00] Copying d/ source...
INFO[2023-07-19T14:46:00+01:00] Finished copying d/ into /run/cos/transition
INFO[2023-07-19T14:46:20+01:00] Applying 'after-upgrade-chroot' hook
INFO[2023-07-19T14:46:20+01:00] Running after-upgrade-chroot hook
INFO[2023-07-19T14:46:20+01:00] rebranding
INFO[2023-07-19T14:46:20+01:00] Setting default grub entry to Kairos
INFO[2023-07-19T14:46:21+01:00] Backing up current active image
INFO[2023-07-19T14:46:21+01:00] Moving /run/initramfs/cos-state/cOS/active.img to /run/initramfs/cos-state/cOS/passive.img
INFO[2023-07-19T14:46:21+01:00] Finished moving /run/initramfs/cos-state/cOS/active.img to /run/initramfs/cos-state/cOS/passive.img
INFO[2023-07-19T14:46:21+01:00] Moving /run/initramfs/cos-state/cOS/transition.img to /run/initramfs/cos-state/cOS/active.img
INFO[2023-07-19T14:46:21+01:00] Finished moving /run/initramfs/cos-state/cOS/transition.img to /run/initramfs/cos-state/cOS/active.img
INFO[2023-07-19T14:46:21+01:00] Applying 'after-upgrade' hook
INFO[2023-07-19T14:46:21+01:00] Running after-upgrade hook
INFO[2023-07-19T14:46:21+01:00] Upgrade completed

As can be seen above, this created a new active.img in the state partition

Reboot, and the new image is applied...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment