Set up microk8s with Cilium for development
Microk8s is a Canonical project to provide a kubernetes environment for local development, similar to minikube but without requiring a separate VM to manage. These instructions describe setting it up for common development use cases with Cilium and may be helpful in particular for testing BPF kernel extensions with Cilium.
Microk8s will run its own containerd runtime, which may be initially confusing when building containers locally with docker. This guide assumes that you will use docker locally for building containers, and push these into a microk8s registry for use by containerd in the microk8s environment.
This guide works with MicroK8s 1.14 or above, with containerd. If you are running an earlier version, see the previous instructions.
- Linux with kernel 4.9 or newer (Full Cilium Requirements)
- Snap (default installed in recent Ubuntu distros)
- docker-ce for new docker client binary, used in local image build.
In this howto setup was run on packet.net c1.small.x86 node with Ubuntu 17.10.
# apt-get install snapd apt-transport-https ca-certificates curl software-properties-common build-essential flex bison clang llvm libelf-dev libssl-dev libcap-dev gcc-multilib libncurses5-dev pkg-config # curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - # add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" # apt-get update # apt-get install docker-ce
Install and set up golang and Cilium
# wget https://dl.google.com/go/go1.11.2.linux-amd64.tar.gz # tar xvf go1.11.2.linux-amd64.tar.gz -C /usr/local/ # mkdir -p ~/go/src/github.com/cilium/
And add to bashrc:
export GOPATH=/<home>/go/ export GOROOT=/usr/local/go/ export PATH=/snap/bin/:/usr/local/go/bin/:/root/go/bin/:$PATH
Then follow with building Cilium itself:
# cd ~/go/src/github.com/cilium/ # git clone https://github.com/cilium/cilium.git && cd cilium/ # go get -u github.com/gordonklaus/ineffassign # go get -u github.com/jteeuwen/go-bindata/... # SKIP_DOCS=true make
Install and set up microk8s and Cilium
The official documentation has a page for Getting Started Using MicroK8s. Follow those instructions to configure your local environment with microk8s for use with Cilium.
# microk8s.enable cilium
Set up other microk8s services
# microk8s.enable dns registry
- The registry is available on localhost:32000 (via NodePort).
Pushing new custom versions of Cilium
Make your local changes to your Cilium repository.
Build and deploy the container image
# make microk8s
This uses your local docker-ce (and docker daemon hosted at
/var/run/docker.sock) to push into the registry that was configured above. It wil pre-pull the custom container image at
LOCAL_IMAGE_TAGis set to something other than
If you have trouble with the above steps, check the Troubleshooting section.
Deploying your new version of Cilium
Per the instructions from the output of
make microk8s, you just need to set the container image in your Cilium DS YAML (or delete the pods to pull a fresh version), then observe that the rollout is successful:
# kubectl -n kube-system set image ds/cilium cilium-agent=localhost:32000/cilium/cilium:local # kubectl -n kube-system rollout status ds cilium
If the tag is already pointing to your custom image, you should just need to delete the pods:
# kubectl -n kube-system delete po -l k8s-app=cilium # kubectl -n kube-system rollout status ds cilium
If the rollout gets stuck it can be debugged through ...
# kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE [...] kube-system cilium-hbcs8 0/1 ErrImagePull 0 75s 184.108.40.206 test <none> [...] # kubectl describe pod -n kube-system cilium-hbcs8 [...] Warning Failed 30s kubelet, test Failed to pull image "localhost:32000/cilium/cilium:my-image": rpc error: code = Unknown desc = Error while pulling image: Get http://localhost:32000/v1/repositories/cilium/cilium/images: read tcp localhost:53302->127.0.0.1:32000: read: connection reset by peer Normal BackOff 4s (x4 over 103s) kubelet, test Back-off pulling image "localhost:32000/cilium/cilium:my-image" Warning Failed 4s (x4 over 103s) kubelet, test Error: ImagePullBackOff [...]
... e.g. in this case the
imagePullPolicy was probably set to
The daemon set updates are undone via:
# kubectl -n kube-system rollout undo ds cilium
Check if Cilium is up and running:
# kubectl get pods --all-namespaces -o wide # kubectl -n kube-system logs --timestamps cilium-1234
Deploying a sample application for testing Cilium w/o policy first:
# kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/minikube/http-sw-app.yaml # kubectl exec -it -n kube-system cilium-1234 -- cilium endpoint list # kubectl exec -it tiefighter -- netperf -t TCP_STREAM -H 10.23.177.124 [...]
Force endpoint regeneration:
# kubectl delete po tiefighter
Running Ginkgo CI (e2e) tests via microk8s
First, add the
cilium.io/ci-node=k8s1 labels to your node:
$ kubectl label node $(hostname) cilium.io/ci-node=k8s1
Run specific ginkgo tests (in this case
MonitorAggregation) against the microk8s environment from the
$ CNI_INTEGRATION=microk8s CILIUM_IMAGE="localhost:32000/cilium/cilium:local" CILIUM_OPERATOR_IMAGE="docker.io/cilium/operator:latest" K8S_VERSION=1.16 ginkgo -v -focus="MonitorAggregation*" -- -cilium.provision=false -cilium.kubeconfig=/home/joe/.kube/config -cilium.passCLIEnvironment=true -cilium.testScope=k8s -cilium.holdEnvironment=true -cilium.skipLogs=true
CNI_INTEGRATIONis required to properly configure the Cilium YAMLs for microk8s
CILIUM_IMAGE: Depends on an image built and pushed via
CILIUM_OPERATOR_IMAGE: This is using upstream master operator; if doing operator development, you can manually bulid / push / set this operator image differently.
-focus: Specify which tests to run
-cilium.provision=false: Disable the k8s cluster provisioning
-cilium.kubeconfig: You will need to update this to point to your own kubeconfig.
-cilium.passCLIEnvironment=true: Necessary to run CI tests in custom cluster (like microk8s)
-cilium.testScope=k8s: Only run the tests in the kubernetes scope
-cilium.holdEnvironment=true: If the test fails, stop everything so the developer can debug the environment in the failed state
-cilium.skipLogs=true: Disable log-gathering at the end of a failed test.
When the test is finished, this will remove cilium from your microk8s environment (as of October 2019). If this bothers you, please send a patch upstream to avoid this.
Some CI tests are written to run against multiple nodes. As of October 2019, these will not successfully pass on microk8s.
General microk8s troubleshooting steps may reveal the problem. A few common issues are also documented below.
Kubectl can't reach apiserver: TLS handshake timeout
Unable to connect to the server: net/http: TLS handshake timeout
microk8s.kubectl instance uses
/var/snap/microk8s/current/credentials/client.config for the password and secrets for connecting to the local apiserver. You can craft these into your own
~/.kube/config file to allow your regular kubectl to access microk8s as well. Note that if you
microk8s.reset or uninstall/reinstall microk8s, you will need to repeat this process as microk8s will generate new secrets.
Prepull gets stuck
$ microk8s.kubectl -n kube-system get pods ... kube-system pod/prepull-dldt5 0/1 Init 0 26m 10.29.161.126 ... <none> <none>
Make sure that you have an entry in
/etc/hosts for your hostname:
$ echo "127.0.0.1 $(hostname)" >> /etc/hosts
Try restarting kubelet:
$ systemctl restart snap.microk8s.daemon-kubelet.service
If this doesn't work, you can try changing the cilium image back to a stable tag. Sometimes this fixes up the prepull pod's connectivity to allow it to pull your new local image, and then you can continue to actually configure Cilium to the new local developer image tag.
$ microk8s.kubectl -n kube-system set image daemonset/cilium cilium-agent=docker.io/cilium/cilium:stable
Prepull gets stuck with no available registry endpoint
$ microk8s.kubectl -n kube-system describe pod -l name=prepull ... Failed to pull image "127.0.0.1:32000/cilium/cilium:local": rpc error: code = Unknown desc = failed to resolve image "127.0.0.1:32000/cilium/cilium:local": no available registry endpoint: failed to do request: Head https://127.0.0.1:32000/v2/cilium/cilium/manifests/local: http: server gave HTTP response to HTTPS client
Check that the registry is working correctly. Is it listening on a socket locally?
$ ss -ta | grep 32000 LISTEN 0 128 *:32000 *:*
Is cilium working properly?
$ microk8s.cilium status ... $ microk8s.kubectl get pods -n kube-system ...
Cannot push Cilium image via
$ make microk8s ... DEPLOY image to microk8s (localhost:32000/cilium/cilium:local) docker tag cilium/cilium-dev:local localhost:32000/cilium/cilium:local docker push localhost:32000/cilium/cilium:local The push refers to repository [localhost:32000/cilium/cilium] Get http://localhost:32000/v2/: dial tcp 127.0.0.1:32000: connect: connection refused
This suggests that the registry is not available locally. Ensure that it has been enabled via:
Then ensure that the registry pods are ready:
$ microk8s.kubectl get pods -n container-registry NAMESPACE NAME READY STATUS RESTARTS AGE container-registry registry-d7d7c8bc9-cwjgt 1/1 Ready 0 17m
If the pods aren't ready, investigate further with
kubectl describe pod .... Ensure that cilium is enabled.
Cilium pods cannot be deployed due to cluster policy
This error occurs when deploying Cilium:
Forbidden: disallowed by cluster policy microk8s
Resolution: Ensure that the microk8s kube-apiserver is configured with
/var/snap/microk8s/current/args/kube-apiserver and microk8s has been restarted with this option configured (
Cilium EOF when attempting to reach docker
Ensure that the docker socket has been updated in the Cilium-DS YAML:
In case there is a change in host IP, you can restart kubernetes API server the following way in order to propagte the new IP to all kubernetes cluster members:
# microk8s.stop # microk8s.start
Problems pushing to docker registry
The following error may occur when IP connectivity to the registry is not available:
# docker push localhost:32000/cilium/cilium:my-image The push refers to repository [localhost:32000/cilium/cilium] Get http://localhost:32000/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
This may occur for multiple reasons:
- Your exernal IP has changed (for instance, your laptop was suspended and restored on a new network)
- For this case, you can follow the instructions above to Restart microk8s.
- Your localhost attempts to connect to the registry via IPv6
- The docker daemon is not listening on IPv4, so if the docker destination of
localhost:32000resolves to IPv6 this may cause timeouts. To overcome this, use
localhost, or remove the IPv6 (
::1) host alias from
- The docker daemon is not listening on IPv4, so if the docker destination of
Some quick-links for general troubleshooting: