Set up microk8s with Cilium for development
Microk8s is a Canonical project to provide a kubernetes environment for local development, similar to minikube but without requiring a separate VM to manage. These instructions describe setting it up for common development use cases with Cilium and may be helpful in particular for testing BPF kernel extensions with Cilium.
Microk8s will run its own containerd runtime, which may be initially confusing when building containers locally with docker. This guide assumes that you will use docker locally for building containers, and push these into a microk8s registry for use by containerd in the microk8s environment.
This guide works with MicroK8s 1.14 or above, with containerd. If you are running an earlier version, see the previous instructions.
- Linux with kernel 4.9 or newer (Full Cilium Requirements)
- Snap (default installed in recent Ubuntu distros)
- docker-ce for new docker client binary, used in local image build.
In this howto setup was run on packet.net c1.small.x86 node with Ubuntu 17.10.
# apt-get install snapd apt-transport-https ca-certificates curl software-properties-common build-essential flex bison clang llvm libelf-dev libssl-dev libcap-dev gcc-multilib libncurses5-dev pkg-config # curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - # add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" # apt-get update # apt-get install docker-ce
Install and set up golang and Cilium
# wget https://dl.google.com/go/go1.11.2.linux-amd64.tar.gz # tar xvf go1.11.2.linux-amd64.tar.gz -C /usr/local/ # mkdir -p ~/go/src/github.com/cilium/
And add to bashrc:
export GOPATH=/<home>/go/ export GOROOT=/usr/local/go/ export PATH=/snap/bin/:/usr/local/go/bin/:/root/go/bin/:$PATH
Then follow with building Cilium itself:
# cd ~/go/src/github.com/cilium/ # git clone https://github.com/cilium/cilium.git && cd cilium/ # go get -u github.com/gordonklaus/ineffassign # go get -u github.com/jteeuwen/go-bindata/... # SKIP_DOCS=true make
Install and set up μK8s and Cilium
The official documentation has a page for Getting Started Using MicroK8s. Follow those instructions to configure your local environment with microk8s for use with Cilium.
Set up other μK8s services
# microk8s.enable dns registry
- The registry is available on localhost:32000 (via NodePort).
Pushing new custom versions of Cilium
Make your local changes to your Cilium repository.
Build and deploy the container image
# make microk8s
This uses your local docker-ce (and docker daemon hosted at
/var/run/docker.sock) to push into the registry that was configured above. It wil pre-pull the custom container image at
LOCAL_IMAGE_TAGis set to something other than
If you have trouble with the above steps, check the Troubleshooting section.
Deploying your new version of Cilium
Per the instructions from the output of
make microk8s, you just need to set the container image in your Cilium DS YAML (or delete the pods to pull a fresh version), then observe that the rollout is successful:
# kubectl -n kube-system set image ds/cilium cilium-agent=localhost:32000/cilium/cilium:local # kubectl -n kube-system rollout status ds cilium
If the tag is already pointing to your custom image, you should just need to delete the pods:
# kubectl -n kube-system delete po -l k8s-app=cilium # kubectl -n kube-system rollout status ds cilium
If the rollout gets stuck it can be debugged through ...
# kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE [...] kube-system cilium-hbcs8 0/1 ErrImagePull 0 75s 188.8.131.52 test <none> [...] # kubectl describe pod -n kube-system cilium-hbcs8 [...] Warning Failed 30s kubelet, test Failed to pull image "localhost:32000/cilium/cilium:my-image": rpc error: code = Unknown desc = Error while pulling image: Get http://localhost:32000/v1/repositories/cilium/cilium/images: read tcp localhost:53302->127.0.0.1:32000: read: connection reset by peer Normal BackOff 4s (x4 over 103s) kubelet, test Back-off pulling image "localhost:32000/cilium/cilium:my-image" Warning Failed 4s (x4 over 103s) kubelet, test Error: ImagePullBackOff [...]
... e.g. in this case the
imagePullPolicy was probably set to
The daemon set updates are undone via:
# kubectl -n kube-system rollout undo ds cilium
Check if Cilium is up and running:
# kubectl get pods --all-namespaces -o wide # kubectl -n kube-system logs --timestamps cilium-1234
Deploying a sample application for testing Cilium w/o policy first:
# kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/minikube/http-sw-app.yaml # kubectl exec -it -n kube-system cilium-1234 -- cilium endpoint list # kubectl exec -it tiefighter -- netperf -t TCP_STREAM -H 10.23.177.124 [...]
Force endpoint regeneration:
# kubectl delete po tiefighter
Running Ginkgo CI (e2e) tests via microk8s
First, add the
cilium.io/ci-node=k8s1 labels to your node:
$ kubectl edit node $(hostname)
.metadata.labels, add the label from the above.
Run specific ginkgo tests (in this case
MonitorAggregation) against the microk8s environment from the
$ CNI_INTEGRATION=microk8s CILIUM_IMAGE="localhost:32000/cilium/cilium:local" CILIUM_OPERATOR_IMAGE="docker.io/cilium/operator:latest" K8S_VERSION=1.16 ginkgo -v -focus="MonitorAggregation*" -- -cilium.provision=false -cilium.kubeconfig=/home/joe/.kube/config -cilium.passCLIEnvironment=true -cilium.testScope=k8s -cilium.holdEnvironment=true -cilium.skipLogs=true
CNI_INTEGRATIONis required to properly configure the Cilium YAMLs for microk8s
CILIUM_IMAGE: Depends on an image built and pushed via
CILIUM_OPERATOR_IMAGE: This is using upstream master operator; if doing operator development, you can manually bulid / push / set this operator image differently.
-focus: Specify which tests to run
-cilium.provision=false: Disable the k8s cluster provisioning
-cilium.kubeconfig: You will need to update this to point to your own kubeconfig.
-cilium.passCLIEnvironment=true: Necessary to run CI tests in custom cluster (like microk8s)
-cilium.testScope=k8s: Only run the tests in the kubernetes scope
-cilium.holdEnvironment=true: If the test fails, stop everything so the developer can debug the environment in the failed state
-cilium.skipLogs=true: Disable log-gathering at the end of a failed test.
When the test is finished, this will remove cilium from your microk8s environment (as of October 2019). If this bothers you, please send a patch upstream to avoid this.
Some CI tests are written to run against multiple nodes. As of October 2019, these will not successfully pass on microk8s.
General microk8s troubleshooting steps may reveal the problem. A few common issues are also documented below.
Cilium EOF when attempting to reach docker
Ensure that the docker socket has been updated in the Cilium-DS YAML:
In case there is a change in host IP, you can restart kubernetes API server the following way in order to propagte the new IP to all kubernetes cluster members:
# microk8s.stop # microk8s.start
Problems pushing to docker registry
The following error may occur when IP connectivity to the registry is not available:
# docker push localhost:32000/cilium/cilium:my-image The push refers to repository [localhost:32000/cilium/cilium] Get http://localhost:32000/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
This may occur for multiple reasons:
- Your exernal IP has changed (for instance, your laptop was suspended and restored on a new network)
- For this case, you can follow the instructions above to Restart μK8s.
- Your localhost attempts to connect to the registry via IPv6
- The docker daemon is not listening on IPv4, so if the docker destination of
localhost:32000resolves to IPv6 this may cause timeouts. To overcome this, use
localhost, or remove the IPv6 (
::1) host alias from
- The docker daemon is not listening on IPv4, so if the docker destination of
Problems deploying Cilium daemon set
When deploying cilium daemon set for the first time, the following error may occur. Make sure to have
--allow-privileged option set in kube-apiserver and apiserver reloaded:
# kubectl create -n kube-system -f https://raw.githubusercontent.com/cilium/cilium/1.3.0/examples/kubernetes/addons/etcd/standalone-etcd.yaml [...] The DaemonSet "cilium" is invalid: * spec.template.spec.containers.securityContext.privileged: Forbidden: disallowed by cluster policy * spec.template.spec.initContainers.securityContext.privileged: Forbidden: disallowed by cluster policy
Some quick-links for general troubleshooting: