Skip to content

Instantly share code, notes, and snippets.

@sameo

sameo/cc-crio-k8s.md

Last active November 1, 2021 23:43
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save sameo/4140e10f55059f7734c851c038ca0eda to your computer and use it in GitHub Desktop.
Save sameo/4140e10f55059f7734c851c038ca0eda to your computer and use it in GitHub Desktop.
Download ZIP
Install and run CC with CRI-O for kubernetes, with kubeadm
Raw
cc-crio-k8s.md

Table of Contents

Guide to install and run a bare metal k8s cluster running on top of Clear Containers and CRI-O, with kubeadm

Requirements

We will be running a bare metal k8s cluster on 2 physical machines. k8s-master will be the master node and k8s-node will be the minion.

Both machines run Ubuntu 16.04.2 server

Install the latest k8s packages

  1. Update your machines
# apt-get update && apt-get install -y apt-transport-https
  1. Use the unstable Ubuntu packages to run k8s 1.6 beta packages (with default CRI support)
# cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial-unstable main
EOF
# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
# apt-get update
  1. Install docker and the k8s packages
# apt-get install -y docker.io kubelet kubeadm kubectl kubernetes-cni

Install, patch, build and start CRI-O

  1. Install all CRI-O dependencies
# apt-get install libseccomp2 libseccomp-dev seccomp libdevmapper-dev libdevmapper1.02.1 libgpgme11 libgpgme11-dev libglib2.0-dev aufs-tools golang-go btrfs-tools
  1. Get the CRI-O code
# mkdir ~/go
# export GOPATH=~/go
# go get github.com/kubernetes-incubator/cri-o
# go get github.com/cpuguy83/go-md2man
  1. Patch and build CRI-O (PR pending)
# cd $GOPATH/src/github.com/kubernetes-incubator/cri-o
# git remote add sameo https://github.com/sameo/cri-o && git fetch sameo
# git pull sameo topic/kubecon-demo
# make && make install
  1. Install the CRI-O systemd service file
# sh -c 'echo "[Unit]
Description=OCI-based implementation of Kubernetes Container Runtime Interface
Documentation=https://github.com/kubernetes-incubator/cri-o

[Service]
ExecStart=/usr/local/bin/ocid --debug
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target" > /etc/systemd/system/ocid.service'
  1. Add the configuration files
# cd $GOPATH/src/github.com/kubernetes-incubator/cri-o
# mkdir /etc/ocid
# mkdir /etc/containers
# cp seccomp.json /etc/ocid/
# cp test/policy.json /etc/containers/
  1. Install the CRI-O configuration file

From the CRI-O configuration file one can decide to run CRI-O with either runc or Clear Containers (COR) as the container runtime.

Using runc as the CRI-O container runtime

# wget https://gist.githubusercontent.com/sameo/4972f1af2f9f7e7a40c7a9a5daa76878/raw/a033a11984549a94810ecc332f21db52afa80386/ocid-runc.conf -O /etc/ocid/ocid.conf

Using COR as the CRI-O container runtime

# wget https://gist.githubusercontent.com/sameo/178398d932fe332f25d894f3ff9b02bb/raw/919fc5d7748298e73609939ee97921fb445228f4/ocid-cor.conf -O /etc/ocid/ocid.conf
  1. Enable and start ocid
# systemctl daemon-reload
# systemctl enable ocid
# systemctl start ocid

Modify kubelet systemd service to use the ocid socket

  1. Create the kubelet systemd directory
# mkdir -p /etc/systemd/system/kubelet.service.d/
  1. Modify the kubelet systemd service
# sh -c 'echo "[Service]
Environment=\"KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true\"
Environment=\"KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true\"
Environment=\"KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin\"
Environment=\"KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local\"
Environment=\"KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt\"
Environment=\"KUBELET_EXTRA_ARGS=--enable-cri --container-runtime=remote --container-runtime-endpoint=/var/run/ocid.sock --runtime-request-timeout=30m\"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_EXTRA_ARGS" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf'
  1. Restart kubelet
# systemctl daemon-reload
# systemctl restart kubelet

Install, patch and build Clear Containers

NOTE: You only need to install Clear Containers if you chose to run CRI-O with Clear Containers (COR) as the container runtime. Otherwise you can skip that section.

First we need to install the Clear Containers 2.1.1 Ubuntu packages and then we'll overwrite them with a modified version.

  1. Install Clear Containers 2.1.1
# sh -c "echo 'deb http://download.opensuse.org/repositories/home:/clearlinux:/preview:/clear-containers-2.1/xUbuntu_$(lsb_release -rs)/ /' >> /etc/apt/sources.list.d/cc-oci-runtime.list"
curl -fsSL http://download.opensuse.org/repositories/home:clearlinux:preview:clear-containers-2.1/xUbuntu_$(lsb_release -rs)/Release.key | sudo apt-key add -
# apt-get update
# apt-get install -y cc-oci-runtime
  1. Install all Clear Containers dependencies
# apt-get install -y build-essential python zlib1g-dev libcap-ng-dev libglib2.0-dev libattr1-dev libcap-dev autoconf libtool libjson-glib-dev uuid-dev check bats libdevmapper-dev file apt-utils libmnl-dev wget git
  1. Get the Clear Containers source code
# go get github.com/01org/cc-oci-runtime
  1. Build and install Clear Containers
# sh ./autogen.sh --disable-static --disable-cppcheck --disable-tests --disable-valgrind-memcheck --disable-functional-tests --disable-docker-tests --with-cc-kernel=/usr/share/clear-containers/vmlinux.container --with-cc-image=/usr/share/clear-containers/clear-containers.img --with-cc-image-systemdsystemunitdir=/usr/lib/systemd/system --enable-autogopath
# make && make install

Start the k8s cluster

We will use kubeadm 1.7 to bring the k8s cluster up.

  1. Start the Clear Containers proxy with full debug

Note: This is not needed if you chose runc as the CRI-O container runtime

# /usr/libexec/cc-proxy -v=4
  1. Initialize the cluster
# kubeadm init --pod-network-cidr 10.244.0.0/16 && export KUBECONFIG=/etc/kubernetes/admin.conf && kubectl apply -f https://gist.githubusercontent.com/feiskyer/1e7a95f27c391a35af47881eb20131d7/raw/4266f05355590fa185bc8e50c0f50d2841993d20/flannel.yaml

Tear the k8s cluster down

Unfortunately, CRI-O is not completely cleaned up when getting called by kubeadm reset. We need to do some additional cleaning.

Both kubeadm reset and the additional CRI-O cleanup needed are combined in one script.

  1. Get the CRI-O cleanup script
# wget https://gist.githubusercontent.com/sameo/de7830848f3a65535f4e9660277f766f/raw/33e100625178df90098df66f391f97802e2eb224/k8s-tear-down.sh
# chmod a+x k8s-tear-down.sh
  1. Tear the k8s cluster down
# ./k8s-tear-down.sh

Debugging

  • To watch runc containers: runc list
  • To get the ocid logs: journalctl -u ocid -n 1000 --no-pager
  • To get the kubelet logs: journalctl -u kubelet -n 1000 --no-pager
  • To get the cc-oci-runtime logs: cat /var/run/cc-oci-runtime.log
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment