Skip to content

Instantly share code, notes, and snippets.

@gnawux

gnawux/Containerd-Kata-shim-v2.md

Created September 15, 2018 03:09
Show Gist options
  • Save gnawux/d06c34b845aa3350799cbeaeb3c1270e to your computer and use it in GitHub Desktop.
Save gnawux/d06c34b845aa3350799cbeaeb3c1270e to your computer and use it in GitHub Desktop.
Download ZIP
run with containerd-Kata-shim-v2
Raw
Containerd-Kata-shim-v2.md

Note: this doc credit to Fupan Li

cri

Note: The standalone cri-containerd binary is end-of-life. cri-containerd is transitioning from a standalone binary that talks to containerd to a plugin within containerd. This github branch is for the cri plugin. See standalone-cri-containerd branch for information about the standalone version of cri-containerd.

Note: You need to drain your node before upgrading from standalone cri-containerd to containerd with cri plugin.

Build Status Go Report Card

cri is a containerd plugin implementation of Kubernetes container runtime interface (CRI).

With it, you could run Kubernetes using containerd as the container runtime. cri

Current Status

cri is a native plugin of containerd 1.1 and above. It is built into containerd and enabled by default.

cri is in GA:

See test dashboard

Support Metrics

CRI-Containerd Version Containerd Version Kubernetes Version CRI Version
v1.0.0-alpha.x 1.7, 1.8 v1alpha1
v1.0.0-beta.x 1.9 v1alpha1
End-Of-Life v1.1 1.10+ v1alpha2
HEAD 1.10+ v1alpha2

Production Quality Cluster on GCE

For a production quality cluster on GCE brought up with kube-up.sh refer here.

Installing with Ansible and Kubeadm

For a multi node cluster installer and bring up steps using ansible and kubeadm refer here.

Custom Installation

For non ansible users, you can download the cri-containerd release tarball and deploy kubernetes cluster using kubeadm as described here.

Getting Started for Developers

Binary Dependencies and Specifications

The current release of the cri plugin has the following dependencies:

See versions of these dependencies cri is tested with.

As containerd and runc move to their respective general availability releases, we will do our best to rebase/retest cri with these releases on a weekly/monthly basis. Similarly, given that cri uses the Open Container Initiative (OCI) image and runtime specifications, we will also do our best to update cri to the latest releases of these specifications as appropriate.

Install Dependencies

  1. Install development libraries:
  • libseccomp development library. Required by cri and runc seccomp support. libseccomp-dev (Ubuntu, Debian) / libseccomp-devel (Fedora, CentOS, RHEL). On releases of Ubuntu <=Trusty and Debian <=jessie a backport version of libseccomp-dev is required. See travis.yml for an example on trusty.
  • btrfs development library. Required by containerd btrfs support. btrfs-tools(Ubuntu, Debian) / btrfs-progs-devel(Fedora, CentOS, RHEL)
  1. Install socat (required by portforward).
  2. Install and setup a go 1.10 development environment.
  3. Make a local clone of this repository.
  4. Install binary dependencies by running the following command from your cloned cri/ project directory:
# Note: install.deps installs the above mentioned runc, containerd, and CNI
# binary dependencies. install.deps is only provided for general use and ease of
# testing. To customize `runc` and `containerd` build tags and/or to configure
# `cni`, please follow instructions in their documents.
make install.deps

Build and Install cri

To build and install a version of containerd with the cri plugin, enter the following commands from your cri project directory:

make
sudo make install

NOTE: The version of containerd built and installed from the Makefile is only for testing purposes. The version tag carries the suffix "-TEST".

Build Tags

cri supports optional build tags for compiling support of various features. To add build tags to the make option the BUILD_TAGS variable must be set.

make BUILD_TAGS='seccomp apparmor'
Build Tag Feature Dependency
seccomp syscall filtering libseccomp development library
selinux selinux process and mount labeling
apparmor apparmor profile support

To use Kata as the runtime for containerd + cri

Build and Install kata-runtime

$ go get -d -u github.com/hyperhq/kata-runtime
$ mv $GOPATH/src/github.com/hyperhq $GOPATH/src/github.com/kata-containers
$ mv $GOPATH/src/github.com/kata-containers/kata-runtime $GOPATH/src/github.com/kata-containers/runtime 
$ cd $GOPATH/src/github.com/kata-containers/runtime
$ git checkout shimv2
$ make && sudo -E PATH=$PATH make install

The build will create the following:

  • runtime binary shim: /usr/local/bin/containerd-shim-kata-v2
  • configuration file: /usr/share/defaults/kata-containers/configuration.toml

Enable the factory template feature by editing the configure file

$ sudo sed -i 's/^#*enable_template/enable_template/' /usr/share/defaults/kata-containers/configuration.toml

Uncoment the image setting, let kata use the default initrd

$ sudo sed -i 's/^image =/#image =/'  /usr/share/defaults/kata-containers/configuration.toml

Create and install kata initrd image

Build a custom Kata agent

$ go get -d -u github.com/kata-containers/agent
$ cd $GOPATH/src/github.com/kata-containers/agent && make

Get the osbuilder

$ go get -d -u github.com/kata-containers/osbuilder

Create a local rootfs for initrd image

$ export ROOTFS_DIR="${GOPATH}/src/github.com/kata-containers/osbuilder/rootfs-builder/rootfs"
$ sudo rm -rf ${ROOTFS_DIR}
$ cd $GOPATH/src/github.com/kata-containers/osbuilder/rootfs-builder
$ sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true ./rootfs.sh alpine'

AGENT_INIT controls if the guest image uses kata agent as the guest init process. When you create an initrd image, always set AGENT_INIT to yes.

Optionally, add your custom agent binary to the rootfs with the following:

$ sudo install -o root -g root -m 0550 -T ../../agent/kata-agent ${ROOTFS_DIR}/sbin/init

Build an initrd image

$ cd $GOPATH/src/github.com/kata-containers/osbuilder/initrd-builder
$ sudo -E AGENT_INIT=yes USE_DOCKER=true ./initrd_builder.sh ${ROOTFS_DIR}'

Install the initrd image

$ commit=$(git log --format=%h -1 HEAD)
$ date=$(date +%Y-%m-%d-%T.%N%z)
$ image="kata-containers-initrd-${date}-${commit}"
$ sudo install -o root -g root -m 0640 -D kata-containers-initrd.img "/usr/share/kata-containers/${image}"
$ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers-initrd.img)

Install guest kernel images for kata

As a prerequisite, you need to install libelf-dev and bc. Otherwise, you will not be able to build the kernel from sources.

$ go get github.com/kata-containers/tests
$ cd $GOPATH/src/github.com/kata-containers/tests/.ci
$ kernel_arch="$(./kata-arch.sh)"
$ kernel_dir="$(./kata-arch.sh --kernel)"
$ tmpdir="$(mktemp -d)"
$ pushd "$tmpdir"
$ curl -L https://raw.githubusercontent.com/kata-containers/packaging/master/kernel/configs/${kernel_dir}_kata_kvm_4.14.x -o .config
$ kernel_version=$(grep "Linux/[${kernel_arch}]*" .config | cut -d' ' -f3 | tail -1)
$ kernel_tar_file="linux-${kernel_version}.tar.xz"
$ kernel_url="https://cdn.kernel.org/pub/linux/kernel/v$(echo $kernel_version | cut -f1 -d.).x/${kernel_tar_file}"
$ curl -LOk ${kernel_url}
$ tar -xf ${kernel_tar_file}
$ mv .config "linux-${kernel_version}"
$ pushd "linux-${kernel_version}"
$ curl -L https://raw.githubusercontent.com/kata-containers/packaging/master/kernel/patches/0001-NO-UPSTREAM-9P-always-use-cached-inode-to-fill-in-v9.patch | patch -p1
$ make ARCH=${kernel_dir} -j$(nproc)
$ kata_kernel_dir="/usr/share/kata-containers"
$ kata_vmlinuz="${kata_kernel_dir}/kata-vmlinuz-${kernel_version}.container"
$ [ $kernel_arch = ppc64le ] && kernel_file="$(realpath ./vmlinux)" || kernel_file="$(realpath arch/${kernel_arch}/boot/bzImage)"
$ sudo install -o root -g root -m 0755 -D "${kernel_file}" "${kata_vmlinuz}"
$ sudo ln -sf "${kata_vmlinuz}" "${kata_kernel_dir}/vmlinuz.container"
$ kata_vmlinux="${kata_kernel_dir}/kata-vmlinux-${kernel_version}"
$ sudo install -o root -g root -m 0755 -D "$(realpath vmlinux)" "${kata_vmlinux}"
$ sudo ln -sf "${kata_vmlinux}" "${kata_kernel_dir}/vmlinux.container"
$ popd
$ popd
$ rm -rf "${tmpdir}"

To use kata-runtime as the containerd runtime for CRI

please set the following in the containerd's config.toml file

[plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.kata.v2"
        runtime_engine = "io.containerd.runtime.kata.v2"

Validate Your cri Setup

A Kubernetes incubator project called cri-tools includes programs for exercising CRI implementations such as the cri plugin. More importantly, cri-tools includes the program critest which is used for running CRI Validation Testing.

Run the CRI Validation test to validate your installation of containerd with cri built in:

make test-cri

Running a Kubernetes local cluster

If you already have a working development environment for supported Kubernetes version, you can try cri in a local cluster:

  1. Start the version of containerd with cri plugin that you built and installed above as root in a first terminal:
sudo containerd
  1. From the Kubernetes project directory startup a local cluster using containerd:
CONTAINER_RUNTIME=remote CONTAINER_RUNTIME_ENDPOINT='unix:///run/containerd/containerd.sock' ./hack/local-up-cluster.sh

Test

See here for information about test.

Using crictl

See here for information about using crictl to debug pods, containers, and images.

Configurations

See here for information about how to configure cri plugins and here for information about how to configure containerd

Documentation

See here for additional documentation.

Contributing

Interested in contributing? Check out the documentation.

Communication

This project was originally established in April of 2017 in the Kubernetes Incubator program. After reaching the Beta stage, In January of 2018, the project was merged into containerd.

For async communication and long running discussions please use issues and pull requests on this github repo. This will be the best place to discuss design and implementation.

For sync communication we have a community slack with a #containerd channel that everyone is welcome to join and chat about development.

Slack: https://dockr.ly/community

Other Communications

As this project is tightly coupled to CRI and CRI-Tools and they are Kubernetes projects, some of our project communications take place in the Kubernetes' SIG: sig-node.

For more information about sig-node, CRI, and the CRI-Tools projects:

Reporting Security Issues

If you are reporting a security issue, please reach out discreetly at security@containerd.io.

Licenses

The containerd codebase is released under the Apache 2.0 license. The README.md file, and files in the "docs" folder are licensed under the Creative Commons Attribution 4.0 International License under the terms and conditions set forth in the file "LICENSE.docs". You may obtain a duplicate copy of the same license, titled CC-BY-4.0, at http://creativecommons.org/licenses/by/4.0/.

Code of Conduct

This project follows the CNCF Code of Conduct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment