Using crio runtime in KIND

README.md

How to use CRI-O runtime with KIND

KIND uses containerd by default as container runtime, however, it is possible to switch it by CRI-O with some modifications

1. Create the new node image, it's based on current KIND images, so the same process applies, you just need to tweak the CRI-O config accordenly (the Dockerfile here may need to be modifies for other k8s versions)

docker build -t kindnode/crio:1.19 .

The image is bigger than the KIND one, of course :-)

REPOSITORY                                             TAG                                 IMAGE ID            CREATED             SIZE
kindnode/crio                                          1.19                                f71390c5d83f        43 minutes ago      1.59GB
kindest/node                                           v1.19.1                             dcaefb48dc5a        40 hours ago        1.36GB

2. With the new image, we just need to create our new cluster with it and patch kubeadm to use the crio socket:

kind create cluster --name crio --image kindnode/crio:1.18 --config kind-config-crio.yaml

and voila, you have a kubernetes cluster using crio as runtime:

kubectl get nodes -o wide
NAME                 STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                     KERNEL-VERSION                CONTAINER-RUNTIME
crio-control-plane   Ready    master   3m12s   v1.18.8   172.19.0.4    <none>        Ubuntu Groovy Gorilla (development branch)   4.18.0-193.6.3.el8_2.x86_64   cri-o://1.18.3
crio-worker          Ready    <none>   2m23s   v1.18.8   172.19.0.2    <none>        Ubuntu Groovy Gorilla (development branch)   4.18.0-193.6.3.el8_2.x86_64   cri-o://1.18.3
crio-worker2         Ready    <none>   2m23s   v1.18.8   172.19.0.3    <none>        Ubuntu Groovy Gorilla (development branch)   4.18.0-193.6.3.el8_2.x86_64   cri-o://1.18.3

3. Install new CRI-O version, since CRI-O is a standalone binary you just need to copy it in each node and restart it:

for n in $(kind get nodes --name crio); do
  docker cp crio $n:/usr/bin/crio
  docker exec $n systemctl restart crio
done

Dockerfile

ARG IMAGE=kindest/node
ARG VERSION=1.19
ARG MINOR=1
ARG OS=xUbuntu_20.04

FROM ${IMAGE}:v${VERSION}.${MINOR}

ARG VERSION
ARG OS

RUN echo "Installing Packages ..." \
    && DEBIAN_FRONTEND=noninteractive clean-install \
    tcpdump \
    vim \
    gnupg \
    tzdata \
 && echo "Installing cri-o" \
    && export CONTAINERS_URL="https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/${OS}/" \
    && echo "deb ${CONTAINERS_URL} /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list \
    && export CRIO_URL="http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/${VERSION}/${OS}/" \
    && echo "deb ${CRIO_URL} /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:${VERSION}.list \
    && curl -L ${CONTAINERS_URL}Release.key | apt-key add - || true \
    && curl -L ${CRIO_URL}Release.key | apt-key add - || true \
    && apt-get update \
    && DEBIAN_FRONTEND=noninteractive apt-get --option=Dpkg::Options::=--force-confdef install -y cri-o cri-o-runc \
    && ln -s /usr/libexec/podman/conmon /usr/local/bin/conmon \
    && sed -i 's/^pause_image =.*/pause_image = \"k8s.gcr.io\/pause:3.2\"/' /etc/crio/crio.conf \
    && sed -i 's/.*storage_driver.*/storage_driver = \"vfs\"/' /etc/crio/crio.conf \
    && sed -i 's/^cgroup_manager =.*/cgroup_manager = \"cgroupfs\"/' /etc/crio/crio.conf \
    && sed -i 's/^cgroup_manager =.*/a conmon_cgroup = \"pod\"/' /etc/crio/crio.conf \
    && sed -i 's/containerd/crio/g' /etc/crictl.yaml \
 && systemctl disable containerd \
 && systemctl enable crio

kind-config-crio.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      criSocket: unix:///run/crio/crio.sock
      kubeletExtraArgs:
        cgroup-driver: cgroupfs
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      criSocket: unix:///run/crio/crio.sock
      kubeletExtraArgs:
        cgroup-driver: cgroupfs
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      criSocket: unix:///run/crio/crio.sock
      kubeletExtraArgs:
        cgroup-driver: cgroupfs

You have to use the crio image

Yes, I'm using currently quay.io/aojea/kindnode:crio1639534040 image and seeing this new issue with kind from @main. Unless you mean by crio image something else.

I see the problem, the crio images are hardcoded to kubernetes 1.21 that doesn't support rootless

https://github.com/aojea/kind-images/blob/f1c63b97dca4dd82476a751f2508d6f043b8f80f/.github/workflows/crio.yaml#L11

In case it's important - I could spin off kind cluster with such another image kindest/node:v1.21.2 with no issues under same setup.

I am trying to run kind with k8s version 1.23 and with crio. Cluster doesn't start properly:

kind-with-crio % kubectl get pods -n kube-system
NAME                                         READY   STATUS                 RESTARTS   AGE
coredns-64897985d-5qks5                      0/1     Running                0          6m28s
coredns-64897985d-df99k                      0/1     Running                0          6m28s
etcd-crio-control-plane                      1/1     Running                0          6m41s
kindnet-gj5vj                                1/1     Running                0          6m28s
kube-apiserver-crio-control-plane            1/1     Running                0          6m41s
kube-controller-manager-crio-control-plane   1/1     Running                0          6m41s
kube-proxy-whlqw                             0/1     CreateContainerError   0          6m28s
kube-scheduler-crio-control-plane            1/1     Running                0          6m41s

coredns stuck in not ready state, logs are full with [INFO] plugin/ready: Still waiting on: "kubernetes"
kube-proxy fails with level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted"
anyone who had the same issue and found a solution here? any ideas here?

I am trying to run kind with k8s version 1.23 and with crio. Cluster doesn't start properly:

kind-with-crio % kubectl get pods -n kube-system
NAME                                         READY   STATUS                 RESTARTS   AGE
coredns-64897985d-5qks5                      0/1     Running                0          6m28s
coredns-64897985d-df99k                      0/1     Running                0          6m28s
etcd-crio-control-plane                      1/1     Running                0          6m41s
kindnet-gj5vj                                1/1     Running                0          6m28s
kube-apiserver-crio-control-plane            1/1     Running                0          6m41s
kube-controller-manager-crio-control-plane   1/1     Running                0          6m41s
kube-proxy-whlqw                             0/1     CreateContainerError   0          6m28s
kube-scheduler-crio-control-plane            1/1     Running                0          6m41s

coredns stuck in not ready state, logs are full with [INFO] plugin/ready: Still waiting on: "kubernetes" kube-proxy fails with level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted" anyone who had the same issue and found a solution here? any ideas here?

at some point it broke , there is some compatibility issue with newer versions and I don't have much time to work on this

I have recently worked in a PoC for testing WasmEdge performance. Initially, I started with CRI-O as runtime manager, using this Dockerfile for the integration. This approach was working with the current 1.24.1 version, but unfortunately I have to abandon it because I have to load some local images with kind load command, and it seems to be tight related to ContainerD. Anyway, I share my discoveries in case that someone find them useful.

I've got it working now using as a base such in Dockerfile:

ARG IMAGE=kindest/node
ARG VERSION=1.24
ARG MINOR=3
ARG OS=xUbuntu_22.04

and having podman 3.4.2 (running it in rootfull way so far - still need to learn more about cgroups v2 for rootless cases) using systemd as cgroupManager and same passed in Dockerfile to cri-o:

printf "[crio.runtime]\ncgroup_manager=\"systemd\"\nconmon_cgroup=\"pod\"\n" > /etc/crio/crio.conf

and was able to create then cluster of 1 control plane node and have that node like such:

$ kubectl --context kind-test123 get nodes -o wide
NAME                    STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION                      CONTAINER-RUNTIME
test123-control-plane   Ready    control-plane   74m   v1.24.3   10.89.0.13    <none>        Ubuntu 22.04.1 LTS   5.4.17-2136.306.1.3.el8uek.x86_64   cri-o://1.24.2

And to test - created httpbin deployment from here: https://github.com/istio/istio/blob/master/samples/httpbin/httpbin.yaml
and did port forwarding via:

kubectl --context kind-test123 port-forward svc/httpbin 8000:8000 -n default

and could hit that with curl calls like such:

curl -X GET localhost:8000/get

And if of interest - here is my crictl images list:

# crictl images
IMAGE                                      TAG                  IMAGE ID            SIZE
docker.io/kennethreitz/httpbin             latest               b138b9264903f       545MB
docker.io/kindest/kindnetd                 v20220726-ed811e41   d921cee849482       63.3MB
docker.io/kindest/local-path-provisioner   v0.0.22-kind.0       4c1e997385b8f       48.9MB
k8s.gcr.io/coredns/coredns                 v1.8.6               a4ca41631cc7a       47MB
k8s.gcr.io/etcd                            3.5.3-0              aebe758cef4cd       301MB
k8s.gcr.io/kube-apiserver                  v1.24.3              d521dd763e2e3       131MB
k8s.gcr.io/kube-controller-manager         v1.24.3              586c112956dfc       121MB
k8s.gcr.io/kube-proxy                      v1.24.3              2ae1ba6417cbc       112MB
k8s.gcr.io/kube-scheduler                  v1.24.3              3a5aa3a515f5d       52.3MB
registry.k8s.io/pause                      3.6                  6270bb605e12e       690kB

So far so good - could play with that then as well in addition to containerd based configurations.

I built a new base image for cri-o. Feel free to give it a try. https://github.com/warm-metal/kindest-base-crio