Skip to content

Instantly share code, notes, and snippets.

@tklauser

tklauser/micro-k8s-setup.md

Forked from joestringer/micro-k8s-setup.md
Last active January 8, 2024 13:37
Show Gist options
  • Save tklauser/1486ee1c375678235cb5035be808745f to your computer and use it in GitHub Desktop.
Save tklauser/1486ee1c375678235cb5035be808745f to your computer and use it in GitHub Desktop.
Download ZIP
MicroK8s development environment setup for Cilium
Raw
micro-k8s-setup.md

Set up microk8s with Cilium for development

Microk8s is a Canonical project to provide a kubernetes environment for local development, similar to minikube but without requiring a separate VM to manage. These instructions describe setting it up for common development use cases with Cilium and may be helpful in particular for testing BPF kernel extensions with Cilium.

Microk8s will run its own containerd runtime, which may be initially confusing when building containers locally with docker. This guide assumes that you will use docker locally for building containers, and push these into a microk8s registry for use by containerd in the microk8s environment.

This guide works with MicroK8s 1.14 or above, with containerd. If you are running an earlier version, see the previous instructions.

Requirements

In this howto setup was run on packet.net c1.small.x86 node with Ubuntu 17.10.

Quick howto:

# apt-get install snapd apt-transport-https ca-certificates curl software-properties-common build-essential flex bison clang llvm libelf-dev libssl-dev libcap-dev gcc-multilib libncurses5-dev pkg-config
# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
# add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
# apt-get update
# apt-get install docker-ce

Install and set up golang and Cilium

# wget https://dl.google.com/go/go1.13.8.linux-amd64.tar.gz
# tar xvf go1.13.8.linux-amd64.tar.gz -C /usr/local/
# mkdir -p ~/go/src/github.com/cilium/

And add to bashrc:

export GOPATH=/<home>/go/
export GOROOT=/usr/local/go/
export PATH=/snap/bin/:/usr/local/go/bin/:/root/go/bin/:$PATH

Then follow with building Cilium itself:

# cd ~/go/src/github.com/cilium/
# git clone https://github.com/cilium/cilium.git && cd cilium/
# go get -u github.com/gordonklaus/ineffassign
# go get -u github.com/jteeuwen/go-bindata/...
# SKIP_DOCS=true make

Install and set up μK8s and Cilium

The official documentation has a page for Getting Started Using MicroK8s. Follow those instructions to configure your local environment with microk8s for use with Cilium.

Optional: add user to microk8s group

This allows to run microk8s commands without sudo.

sudo usermod -a -G microk8s $USER

Optional: set up alias for kubectl command

MicroK8s itself includes a kubectl binary, which can be accessed by running the microk8s.kubectl command.

While using the prefix microk8s.kubectl allows for a parallel install of another system-wide kubectl without impact, you can easily get rid of it by using the snap alias command:

sudo snap alias microk8s.kubectl kubectl

See https://kubernetes.io/blog/2019/11/26/running-kubernetes-locally-on-linux-with-microk8s/ for details.

Set up other μK8s services

# microk8s.enable dns registry
  • The registry is available on localhost:32000 (via NodePort).

Pushing new custom versions of Cilium

  • Make your local changes to your Cilium repository.

  • Build and deploy the container image

    # make microk8s

    This uses your local docker-ce (and docker daemon hosted at /var/run/docker.sock) to push into the registry that was configured above. It wil pre-pull the custom container image at cilium/cilium:local (or cilium/cilium:LOCAL_IMAGE_TAG if LOCAL_IMAGE_TAG is set to something other than local).

If you have trouble with the above steps, check the Troubleshooting section.

Deploying your new version of Cilium

Per the instructions from the output of make microk8s, you just need to set the container image in your Cilium DS YAML (or delete the pods to pull a fresh version), then observe that the rollout is successful:

# kubectl -n kube-system set image ds/cilium cilium-agent=localhost:32000/cilium/cilium:local
# kubectl -n kube-system rollout status ds cilium

If the tag is already pointing to your custom image, you should just need to delete the pods:

# kubectl -n kube-system delete po -l k8s-app=cilium
# kubectl -n kube-system rollout status ds cilium

Rolling back

If the rollout gets stuck it can be debugged through ...

# kubectl get pods --all-namespaces -o wide
NAMESPACE            NAME                                    READY   STATUS         RESTARTS   AGE    IP              NODE   NOMINATED NODE
[...]
kube-system          cilium-hbcs8                            0/1     ErrImagePull   0          75s    147.75.80.23    test   <none>
[...]
# kubectl describe pod -n kube-system cilium-hbcs8
[...]
  Warning  Failed     30s                    kubelet, test      Failed to pull image "localhost:32000/cilium/cilium:my-image": rpc error: code = Unknown desc = Error while pulling image: Get http://localhost:32000/v1/repositories/cilium/cilium/images: read tcp localhost:53302->127.0.0.1:32000: read: connection reset by peer
  Normal   BackOff    4s (x4 over 103s)      kubelet, test      Back-off pulling image "localhost:32000/cilium/cilium:my-image"
  Warning  Failed     4s (x4 over 103s)      kubelet, test      Error: ImagePullBackOff
[...]

... e.g. in this case the imagePullPolicy was probably set to Always.

The daemon set updates are undone via:

# kubectl -n kube-system rollout undo ds cilium

Test workload

Check if Cilium is up and running:

# kubectl get pods --all-namespaces -o wide
# kubectl -n kube-system logs --timestamps cilium-1234

Deploying a sample application for testing Cilium w/o policy first:

# kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/minikube/http-sw-app.yaml
# kubectl exec -it -n kube-system cilium-1234 -- cilium endpoint list
# kubectl exec -it tiefighter -- netperf -t TCP_STREAM -H 10.23.177.124
[...]

Force endpoint regeneration:

# kubectl delete po tiefighter

Running Ginkgo CI (e2e) tests via microk8s

First, add the cilium.io/ci-node=k8s1 labels to your node:

$ kubectl edit node $(hostname)

Under .metadata.labels, add the label from the above.

Run specific ginkgo tests (in this case MonitorAggregation) against the microk8s environment from the test/ directory:

$ CNI_INTEGRATION=microk8s CILIUM_IMAGE="localhost:32000/cilium/cilium:local" CILIUM_OPERATOR_IMAGE="docker.io/cilium/operator:latest" K8S_VERSION=1.16 ginkgo -v -focus="MonitorAggregation*" -- -cilium.provision=false -cilium.kubeconfig=/home/joe/.kube/config -cilium.passCLIEnvironment=true -cilium.testScope=k8s -cilium.holdEnvironment=true -cilium.skipLogs=true

The options:

  • CNI_INTEGRATION is required to properly configure the Cilium YAMLs for microk8s
  • CILIUM_IMAGE: Depends on an image built and pushed via make microk8s
  • CILIUM_OPERATOR_IMAGE: This is using upstream master operator; if doing operator development, you can manually bulid / push / set this operator image differently.
  • -focus: Specify which tests to run
  • -cilium.provision=false: Disable the k8s cluster provisioning
  • -cilium.kubeconfig: You will need to update this to point to your own kubeconfig.
  • -cilium.passCLIEnvironment=true: Necessary to run CI tests in custom cluster (like microk8s)
  • -cilium.testScope=k8s: Only run the tests in the kubernetes scope
  • -cilium.holdEnvironment=true: If the test fails, stop everything so the developer can debug the environment in the failed state
  • -cilium.skipLogs=true: Disable log-gathering at the end of a failed test.

When the test is finished, this will remove cilium from your microk8s environment (as of October 2019). If this bothers you, please send a patch upstream to avoid this.

Limitations

Some CI tests are written to run against multiple nodes. As of October 2019, these will not successfully pass on microk8s.

Troubleshooting

General microk8s troubleshooting steps may reveal the problem. A few common issues are also documented below.

Cilium EOF when attempting to reach docker

Ensure that the docker socket has been updated in the Cilium-DS YAML:

https://gist.github.com/joestringer/60a5f53d59e57274ed4c2a1736a7b101#set-up-cilium

Restart μK8s

In case there is a change in host IP, you can restart kubernetes API server the following way in order to propagte the new IP to all kubernetes cluster members:

# microk8s.stop
# microk8s.start

Problems pushing to docker registry

The following error may occur when IP connectivity to the registry is not available:

# docker push localhost:32000/cilium/cilium:my-image
The push refers to repository [localhost:32000/cilium/cilium]
Get http://localhost:32000/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

This may occur for multiple reasons:

  • Your exernal IP has changed (for instance, your laptop was suspended and restored on a new network)
    • For this case, you can follow the instructions above to Restart μK8s.
  • Your localhost attempts to connect to the registry via IPv6
    • The docker daemon is not listening on IPv4, so if the docker destination of localhost:32000 resolves to IPv6 this may cause timeouts. To overcome this, use 127.0.0.1 instead of localhost, or remove the IPv6 (::1) host alias from /etc/hosts.

Problems deploying Cilium daemon set

When deploying cilium daemon set for the first time, the following error may occur. Make sure to have --allow-privileged option set in kube-apiserver and apiserver reloaded:

# kubectl create -n kube-system -f https://raw.githubusercontent.com/cilium/cilium/1.3.0/examples/kubernetes/addons/etcd/standalone-etcd.yaml
[...]
The DaemonSet "cilium" is invalid: 
* spec.template.spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
* spec.template.spec.initContainers[0].securityContext.privileged: Forbidden: disallowed by cluster policy

Misc

Some quick-links for general troubleshooting:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment