Using crio runtime in KIND

README.md

How to use CRI-O runtime with KIND

KIND uses containerd by default as container runtime, however, it is possible to switch it by CRI-O with some modifications

1. Create the new node image, it's based on current KIND images, so the same process applies, you just need to tweak the CRI-O config accordenly (the Dockerfile here may need to be modifies for other k8s versions)

docker build -t kindnode/crio:1.19 .

The image is bigger than the KIND one, of course :-)

REPOSITORY                                             TAG                                 IMAGE ID            CREATED             SIZE
kindnode/crio                                          1.19                                f71390c5d83f        43 minutes ago      1.59GB
kindest/node                                           v1.19.1                             dcaefb48dc5a        40 hours ago        1.36GB

2. With the new image, we just need to create our new cluster with it and patch kubeadm to use the crio socket:

kind create cluster --name crio --image kindnode/crio:1.18 --config kind-config-crio.yaml

and voila, you have a kubernetes cluster using crio as runtime:

kubectl get nodes -o wide
NAME                 STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                     KERNEL-VERSION                CONTAINER-RUNTIME
crio-control-plane   Ready    master   3m12s   v1.18.8   172.19.0.4    <none>        Ubuntu Groovy Gorilla (development branch)   4.18.0-193.6.3.el8_2.x86_64   cri-o://1.18.3
crio-worker          Ready    <none>   2m23s   v1.18.8   172.19.0.2    <none>        Ubuntu Groovy Gorilla (development branch)   4.18.0-193.6.3.el8_2.x86_64   cri-o://1.18.3
crio-worker2         Ready    <none>   2m23s   v1.18.8   172.19.0.3    <none>        Ubuntu Groovy Gorilla (development branch)   4.18.0-193.6.3.el8_2.x86_64   cri-o://1.18.3

3. Install new CRI-O version, since CRI-O is a standalone binary you just need to copy it in each node and restart it:

for n in $(kind get nodes --name crio); do
  docker cp crio $n:/usr/bin/crio
  docker exec $n systemctl restart crio
done

Dockerfile

ARG IMAGE=kindest/node
ARG VERSION=1.19
ARG MINOR=1
ARG OS=xUbuntu_20.04

FROM ${IMAGE}:v${VERSION}.${MINOR}

ARG VERSION
ARG OS

RUN echo "Installing Packages ..." \
    && DEBIAN_FRONTEND=noninteractive clean-install \
    tcpdump \
    vim \
    gnupg \
    tzdata \
 && echo "Installing cri-o" \
    && export CONTAINERS_URL="https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/${OS}/" \
    && echo "deb ${CONTAINERS_URL} /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list \
    && export CRIO_URL="http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/${VERSION}/${OS}/" \
    && echo "deb ${CRIO_URL} /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:${VERSION}.list \
    && curl -L ${CONTAINERS_URL}Release.key | apt-key add - || true \
    && curl -L ${CRIO_URL}Release.key | apt-key add - || true \
    && apt-get update \
    && DEBIAN_FRONTEND=noninteractive apt-get --option=Dpkg::Options::=--force-confdef install -y cri-o cri-o-runc \
    && ln -s /usr/libexec/podman/conmon /usr/local/bin/conmon \
    && sed -i 's/^pause_image =.*/pause_image = \"k8s.gcr.io\/pause:3.2\"/' /etc/crio/crio.conf \
    && sed -i 's/.*storage_driver.*/storage_driver = \"vfs\"/' /etc/crio/crio.conf \
    && sed -i 's/^cgroup_manager =.*/cgroup_manager = \"cgroupfs\"/' /etc/crio/crio.conf \
    && sed -i 's/^cgroup_manager =.*/a conmon_cgroup = \"pod\"/' /etc/crio/crio.conf \
    && sed -i 's/containerd/crio/g' /etc/crictl.yaml \
 && systemctl disable containerd \
 && systemctl enable crio

kind-config-crio.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      criSocket: unix:///run/crio/crio.sock
      kubeletExtraArgs:
        cgroup-driver: cgroupfs
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      criSocket: unix:///run/crio/crio.sock
      kubeletExtraArgs:
        cgroup-driver: cgroupfs
- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      criSocket: unix:///run/crio/crio.sock
      kubeletExtraArgs:
        cgroup-driver: cgroupfs

Hi @aojea - I'm trying to use one of latest automatically built image quay.io/aojea/kindnode:crio1639534040 and getting such error:

[FAILED] Failed to start Container Runtime Interface for OCI (CRI-O).
See 'systemctl status crio.service' for details.

and while navigating into that running container and checking status - there are such lines:

Dec 15 23:51:58 crio-control-plane crio[147]: time="2021-12-15 23:51:58.298625001Z" level=fatal msg="Validating root config: failed to get store to set defaults: kernel does not support overlay fs: 'overlay' is not supported over xfs at \"/var/lib/containers/storage/overlay\": backing file system is unsupported for this graph driver"
Dec 15 23:51:58 crio-control-plane systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE
Dec 15 23:51:58 crio-control-plane systemd[1]: crio.service: Failed with result 'exit-code'.
Dec 15 23:51:58 crio-control-plane systemd[1]: Failed to start Container Runtime Interface for OCI (CRI-O).

But once I uncomment this line inside /etc/containers/storage.conf file:

#mount_program = "/usr/bin/fuse-overlayfs"

then at least I could start crio via systemcl start crio.service.

Not sure if this is right place to ask such question about your prepared image, but curious what else should I try to check/change? I'm exploring kind+rootless podman setup of things.

we are going to merge a fix in kind for rootless that seems related

kubernetes-sigs/kind#2559

for rootless wait until that PR is merged and use kind latest version from @main

@aojea - is there some snapshot folder to download @main kind binary from? Or only way is to clone that repo and try to build locally?

go install sigs.k8s.io/kind@main

@aojea - installed such kind version from main: go: downloading sigs.k8s.io/kind v0.11.2-0.20211216085318-c88b8ec95949
But with it - using all other same things/image as yesterday - getting into such earlier failure during kind cluster creation:

 ✗ Writing configuration 📜 
ERROR: failed to create cluster: failed to generate kubeadm config content: version "1.21.1" is not compatible with rootless provider (hint: kind v0.11.x may work with this version)

You have to use the crio image

Yes, I'm using currently quay.io/aojea/kindnode:crio1639534040 image and seeing this new issue with kind from @main. Unless you mean by crio image something else.

I see the problem, the crio images are hardcoded to kubernetes 1.21 that doesn't support rootless

https://github.com/aojea/kind-images/blob/f1c63b97dca4dd82476a751f2508d6f043b8f80f/.github/workflows/crio.yaml#L11

In case it's important - I could spin off kind cluster with such another image kindest/node:v1.21.2 with no issues under same setup.

I am trying to run kind with k8s version 1.23 and with crio. Cluster doesn't start properly:

kind-with-crio % kubectl get pods -n kube-system
NAME                                         READY   STATUS                 RESTARTS   AGE
coredns-64897985d-5qks5                      0/1     Running                0          6m28s
coredns-64897985d-df99k                      0/1     Running                0          6m28s
etcd-crio-control-plane                      1/1     Running                0          6m41s
kindnet-gj5vj                                1/1     Running                0          6m28s
kube-apiserver-crio-control-plane            1/1     Running                0          6m41s
kube-controller-manager-crio-control-plane   1/1     Running                0          6m41s
kube-proxy-whlqw                             0/1     CreateContainerError   0          6m28s
kube-scheduler-crio-control-plane            1/1     Running                0          6m41s

coredns stuck in not ready state, logs are full with [INFO] plugin/ready: Still waiting on: "kubernetes"
kube-proxy fails with level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted"
anyone who had the same issue and found a solution here? any ideas here?

I am trying to run kind with k8s version 1.23 and with crio. Cluster doesn't start properly:

kind-with-crio % kubectl get pods -n kube-system
NAME                                         READY   STATUS                 RESTARTS   AGE
coredns-64897985d-5qks5                      0/1     Running                0          6m28s
coredns-64897985d-df99k                      0/1     Running                0          6m28s
etcd-crio-control-plane                      1/1     Running                0          6m41s
kindnet-gj5vj                                1/1     Running                0          6m28s
kube-apiserver-crio-control-plane            1/1     Running                0          6m41s
kube-controller-manager-crio-control-plane   1/1     Running                0          6m41s
kube-proxy-whlqw                             0/1     CreateContainerError   0          6m28s
kube-scheduler-crio-control-plane            1/1     Running                0          6m41s

coredns stuck in not ready state, logs are full with [INFO] plugin/ready: Still waiting on: "kubernetes" kube-proxy fails with level=error msg="container_linux.go:380: starting container process caused: apply caps: operation not permitted" anyone who had the same issue and found a solution here? any ideas here?

at some point it broke , there is some compatibility issue with newer versions and I don't have much time to work on this

I have recently worked in a PoC for testing WasmEdge performance. Initially, I started with CRI-O as runtime manager, using this Dockerfile for the integration. This approach was working with the current 1.24.1 version, but unfortunately I have to abandon it because I have to load some local images with kind load command, and it seems to be tight related to ContainerD. Anyway, I share my discoveries in case that someone find them useful.

I've got it working now using as a base such in Dockerfile:

ARG IMAGE=kindest/node
ARG VERSION=1.24
ARG MINOR=3
ARG OS=xUbuntu_22.04

and having podman 3.4.2 (running it in rootfull way so far - still need to learn more about cgroups v2 for rootless cases) using systemd as cgroupManager and same passed in Dockerfile to cri-o:

printf "[crio.runtime]\ncgroup_manager=\"systemd\"\nconmon_cgroup=\"pod\"\n" > /etc/crio/crio.conf

and was able to create then cluster of 1 control plane node and have that node like such:

$ kubectl --context kind-test123 get nodes -o wide
NAME                    STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION                      CONTAINER-RUNTIME
test123-control-plane   Ready    control-plane   74m   v1.24.3   10.89.0.13    <none>        Ubuntu 22.04.1 LTS   5.4.17-2136.306.1.3.el8uek.x86_64   cri-o://1.24.2

And to test - created httpbin deployment from here: https://github.com/istio/istio/blob/master/samples/httpbin/httpbin.yaml
and did port forwarding via:

kubectl --context kind-test123 port-forward svc/httpbin 8000:8000 -n default

and could hit that with curl calls like such:

curl -X GET localhost:8000/get

And if of interest - here is my crictl images list:

# crictl images
IMAGE                                      TAG                  IMAGE ID            SIZE
docker.io/kennethreitz/httpbin             latest               b138b9264903f       545MB
docker.io/kindest/kindnetd                 v20220726-ed811e41   d921cee849482       63.3MB
docker.io/kindest/local-path-provisioner   v0.0.22-kind.0       4c1e997385b8f       48.9MB
k8s.gcr.io/coredns/coredns                 v1.8.6               a4ca41631cc7a       47MB
k8s.gcr.io/etcd                            3.5.3-0              aebe758cef4cd       301MB
k8s.gcr.io/kube-apiserver                  v1.24.3              d521dd763e2e3       131MB
k8s.gcr.io/kube-controller-manager         v1.24.3              586c112956dfc       121MB
k8s.gcr.io/kube-proxy                      v1.24.3              2ae1ba6417cbc       112MB
k8s.gcr.io/kube-scheduler                  v1.24.3              3a5aa3a515f5d       52.3MB
registry.k8s.io/pause                      3.6                  6270bb605e12e       690kB

So far so good - could play with that then as well in addition to containerd based configurations.

I built a new base image for cri-o. Feel free to give it a try. https://github.com/warm-metal/kindest-base-crio