Build images directly into the K3s containerd instance with BuildKit

Build images directly into the K3s containerd instance with BuildKit.md

I wanted to iterate on some images and have them available to K3s immediately, much like the Docker Desktop experience.

I also wanted to use docker buildx bake for this, but buildctl and nerdctl build should work fine too. You can omit Docker components if you don't want Buildx.

• Ubuntu 22.04
• K3s latest (v1.24.2+k3s1)
• BuildKit v0.10.3
• Docker Engine 20.10.17 (required for Buildx)
• Docker Buildx master (required for new 'remote' driver)
• Nerdctl 0.21.0

# Install K3s
curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=latest sh -s -

# Install BuildKit
curl -L https://github.com/moby/buildkit/releases/download/v0.10.3/buildkit-v0.10.3.linux-amd64.tar.gz \
| tar xzf - -C /usr/local/

# Install nerdctl
curl -L https://github.com/containerd/nerdctl/releases/download/v0.21.0/nerdctl-0.21.0-linux-amd64.tar.gz \
| tar xzf - -C /usr/local/bin

# Install buildkitd.service (below) into /etc/systemd/system/buildkit.service
systemctl daemon-reload
systemctl enable --now buildkitd.service

# Set up Docker + Buildx (skip this if you prefer to just use buildctl)
# Install Docker (https://docs.docker.com/engine/install/ubuntu/)

# Install Buildx (need the released version to build master)
mkdir -p /usr/local/lib/docker/cli-plugins
curl -Lo /usr/local/lib/docker/cli-plugins/docker-buildx \
  https://github.com/docker/buildx/releases/download/v0.8.2/buildx-v0.8.2.linux-amd64
chmod +x /usr/local/lib/docker/cli-plugins/docker-buildx

# Build Buildx master
docker buildx bake https://github.com/docker/buildx.git
mv ./bin/docker-buildx /usr/local/lib/docker/cli-plugins/docker-buildx

# Create a Buildx context
docker buildx create --name k3s --driver remote unix:///run/buildkit/buildkitd.sock --use

# Build something with Buildx
docker buildx build -t docker.io/some-namespace/some-image . --output type=image,unpack=true

# Manage built images with nerdctl
nerdctl --address /run/k3s/containerd/container.sock --namespace k8s.io image (ls/rm/inspect)

⚠️ Important arguments when building images

I found that if I omitted the docker.io part of the image tag and just built an image as some-namespace/some-image, it failed to find the image (or tried to pull it) when I tried to run it in K3s.

I also found that the unpack=true was required in the --output option to ensure the image was fully resolved. Without it, ctr image check showed them as 'unavailable' and they wouldn't run in K3s.

buildkitd.service

[Unit]
Description=BuildKit
Wants=k3s.service
After=k3s.service

[Install]
WantedBy=multi-user.target

[Service]
Restart=always
RestartSec=5s
ExecStart=/usr/local/bin/buildkitd \
  --oci-worker=false \
  --containerd-worker-addr=/run/k3s/containerd/containerd.sock \
  --containerd-worker-namespace=k8s.io

I initially tried running buildkitd in a privileged pod, but it seemed to have problems actually building things. I just settled on running it on the host with systemd.

This is the Deployment that I last tried with in case wants to figure it out...

apiVersion: apps/v1
kind: Deployment
metadata:
  name: buildkit
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: buildkit
  template:
    metadata:
      labels:
        app: buildkit
    spec:
      containers:
        - name: buildkit
          image: moby/buildkit:v0.10.3
          args:
            - --oci-worker=false
            - --containerd-worker-addr=/run/k3s/containerd/containerd.sock
            - --containerd-worker-namespace=k8s.io
            - --group=999
            - --debug
          env:
            - name: PATH
              value: /var/lib/rancher/k3s/data/current/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /run
              name: run
            - mountPath: /var/lib/buildkit
              name: buildkit
            - mountPath: /var/lib/rancher
              name: rancher
            - mountPath: /tmp
              name: tmp
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: buildkit
          hostPath:
            path: /var/lib/buildkit
        - name: rancher
          hostPath:
            path: /var/lib/rancher
        - name: tmp
          hostPath:
            path: /tmp

The method with just Buildkit and no Docker is not working for me (maybe something in buildkit changed) with K3s. I have documented my method here:
moby/buildkit#3629