Skip to content

Instantly share code, notes, and snippets.

@joestringer

joestringer/micro-k8s-setup.md

Last active April 20, 2021 17:06
Show Gist options
  • Star 5 You must be signed in to star a gist
  • Fork 3 You must be signed in to fork a gist
  • Save joestringer/60a5f53d59e57274ed4c2a1736a7b101 to your computer and use it in GitHub Desktop.
Save joestringer/60a5f53d59e57274ed4c2a1736a7b101 to your computer and use it in GitHub Desktop.
Download ZIP
MicroK8s development environment setup for Cilium
Raw
micro-k8s-setup.md

Set up microk8s with Cilium for development

Microk8s is a Canonical project to provide a kubernetes environment for local development, similar to minikube but without requiring a separate VM to manage. These instructions describe setting it up for common development use cases with Cilium and may be helpful in particular for testing BPF kernel extensions with Cilium.

Microk8s will run its own containerd runtime, which may be initially confusing when building containers locally with docker. This guide assumes that you will use docker locally for building containers, and push these into a microk8s registry for use by containerd in the microk8s environment.

This guide works with MicroK8s 1.14 or above, with containerd. If you are running an earlier version, see the previous instructions.

Requirements

In this howto setup was run on packet.net c1.small.x86 node with Ubuntu 17.10.

Quick howto:

# apt-get install snapd apt-transport-https ca-certificates curl software-properties-common build-essential flex bison clang llvm libelf-dev libssl-dev libcap-dev gcc-multilib libncurses5-dev pkg-config
# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
# add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
# apt-get update
# apt-get install docker-ce

Install and set up golang and Cilium

# wget https://dl.google.com/go/go1.11.2.linux-amd64.tar.gz
# tar xvf go1.11.2.linux-amd64.tar.gz -C /usr/local/
# mkdir -p ~/go/src/github.com/cilium/

And add to bashrc:

export GOPATH=/<home>/go/
export GOROOT=/usr/local/go/
export PATH=/snap/bin/:/usr/local/go/bin/:/root/go/bin/:$PATH

Then follow with building Cilium itself:

# cd ~/go/src/github.com/cilium/
# git clone https://github.com/cilium/cilium.git && cd cilium/
# go get -u github.com/gordonklaus/ineffassign
# go get -u github.com/jteeuwen/go-bindata/...
# SKIP_DOCS=true make

Install and set up microk8s and Cilium

The official documentation has a page for Getting Started Using MicroK8s. Follow those instructions to configure your local environment with microk8s for use with Cilium.

# microk8s.enable cilium

Set up other microk8s services

# microk8s.enable dns registry
  • The registry is available on localhost:32000 (via NodePort).

Pushing new custom versions of Cilium

  • Make your local changes to your Cilium repository.

  • Build and deploy the container image

    # make microk8s

    This uses your local docker-ce (and docker daemon hosted at /var/run/docker.sock) to push into the registry that was configured above. It wil pre-pull the custom container image at cilium/cilium:local (or cilium/cilium:LOCAL_IMAGE_TAG if LOCAL_IMAGE_TAG is set to something other than local).

If you have trouble with the above steps, check the Troubleshooting section.

Deploying your new version of Cilium

Per the instructions from the output of make microk8s, you just need to set the container image in your Cilium DS YAML (or delete the pods to pull a fresh version), then observe that the rollout is successful:

# kubectl -n kube-system set image ds/cilium cilium-agent=localhost:32000/cilium/cilium:local
# kubectl -n kube-system rollout status ds cilium

If the tag is already pointing to your custom image, you should just need to delete the pods:

# kubectl -n kube-system delete po -l k8s-app=cilium
# kubectl -n kube-system rollout status ds cilium

Rolling back

If the rollout gets stuck it can be debugged through ...

# kubectl get pods --all-namespaces -o wide
NAMESPACE            NAME                                    READY   STATUS         RESTARTS   AGE    IP              NODE   NOMINATED NODE
[...]
kube-system          cilium-hbcs8                            0/1     ErrImagePull   0          75s    147.75.80.23    test   <none>
[...]
# kubectl describe pod -n kube-system cilium-hbcs8
[...]
  Warning  Failed     30s                    kubelet, test      Failed to pull image "localhost:32000/cilium/cilium:my-image": rpc error: code = Unknown desc = Error while pulling image: Get http://localhost:32000/v1/repositories/cilium/cilium/images: read tcp localhost:53302->127.0.0.1:32000: read: connection reset by peer
  Normal   BackOff    4s (x4 over 103s)      kubelet, test      Back-off pulling image "localhost:32000/cilium/cilium:my-image"
  Warning  Failed     4s (x4 over 103s)      kubelet, test      Error: ImagePullBackOff
[...]

... e.g. in this case the imagePullPolicy was probably set to Always.

The daemon set updates are undone via:

# kubectl -n kube-system rollout undo ds cilium

Test workload

Check if Cilium is up and running:

# kubectl get pods --all-namespaces -o wide
# kubectl -n kube-system logs --timestamps cilium-1234

Deploying a sample application for testing Cilium w/o policy first:

# kubectl create -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/minikube/http-sw-app.yaml
# kubectl exec -it -n kube-system cilium-1234 -- cilium endpoint list
# kubectl exec -it tiefighter -- netperf -t TCP_STREAM -H 10.23.177.124
[...]

Force endpoint regeneration:

# kubectl delete po tiefighter

Running Ginkgo CI (e2e) tests via microk8s

First, add the cilium.io/ci-node=k8s1 labels to your node:

$ kubectl label node $(hostname) cilium.io/ci-node=k8s1

Run specific ginkgo tests (in this case MonitorAggregation) against the microk8s environment from the test/ directory:

$ CNI_INTEGRATION=microk8s CILIUM_IMAGE="localhost:32000/cilium/cilium:local" CILIUM_OPERATOR_IMAGE="docker.io/cilium/operator:latest" K8S_VERSION=1.16 ginkgo -v -focus="MonitorAggregation*" -- -cilium.provision=false -cilium.kubeconfig=/home/joe/.kube/config -cilium.passCLIEnvironment=true -cilium.testScope=k8s -cilium.holdEnvironment=true -cilium.skipLogs=true

The options:

  • CNI_INTEGRATION is required to properly configure the Cilium YAMLs for microk8s
  • CILIUM_IMAGE: Depends on an image built and pushed via make microk8s
  • CILIUM_OPERATOR_IMAGE: This is using upstream master operator; if doing operator development, you can manually bulid / push / set this operator image differently.
  • -focus: Specify which tests to run
  • -cilium.provision=false: Disable the k8s cluster provisioning
  • -cilium.kubeconfig: You will need to update this to point to your own kubeconfig.
  • -cilium.passCLIEnvironment=true: Necessary to run CI tests in custom cluster (like microk8s)
  • -cilium.testScope=k8s: Only run the tests in the kubernetes scope
  • -cilium.holdEnvironment=true: If the test fails, stop everything so the developer can debug the environment in the failed state
  • -cilium.skipLogs=true: Disable log-gathering at the end of a failed test.

When the test is finished, this will remove cilium from your microk8s environment (as of October 2019). If this bothers you, please send a patch upstream to avoid this.

Limitations

Some CI tests are written to run against multiple nodes. As of October 2019, these will not successfully pass on microk8s.

Troubleshooting

General microk8s troubleshooting steps may reveal the problem. A few common issues are also documented below.

Kubectl can't reach apiserver: TLS handshake timeout

Unable to connect to the server: net/http: TLS handshake timeout

The microk8s.kubectl instance uses /var/snap/microk8s/current/credentials/client.config for the password and secrets for connecting to the local apiserver. You can craft these into your own ~/.kube/config file to allow your regular kubectl to access microk8s as well. Note that if you microk8s.reset or uninstall/reinstall microk8s, you will need to repeat this process as microk8s will generate new secrets.

Prepull gets stuck

$ microk8s.kubectl -n kube-system get pods
...
kube-system          pod/prepull-dldt5                           0/1     Init   0          26m     10.29.161.126   ...   <none>           <none>

Make sure that you have an entry in /etc/hosts for your hostname:

$ echo "127.0.0.1 $(hostname)" >> /etc/hosts

Try restarting kubelet:

$ systemctl restart snap.microk8s.daemon-kubelet.service

If this doesn't work, you can try changing the cilium image back to a stable tag. Sometimes this fixes up the prepull pod's connectivity to allow it to pull your new local image, and then you can continue to actually configure Cilium to the new local developer image tag.

$ microk8s.kubectl -n kube-system set image daemonset/cilium cilium-agent=docker.io/cilium/cilium:stable

Prepull gets stuck with no available registry endpoint

$ microk8s.kubectl -n kube-system describe pod -l name=prepull
...
Failed to pull image "127.0.0.1:32000/cilium/cilium:local": rpc error: code = Unknown desc = failed to resolve image "127.0.0.1:32000/cilium/cilium:local": no available registry endpoint: failed to do request: Head https://127.0.0.1:32000/v2/cilium/cilium/manifests/local: http: server gave HTTP response to HTTPS client

Check that the registry is working correctly. Is it listening on a socket locally?

$ ss -ta | grep 32000
LISTEN     0      128        *:32000                    *:*

Is cilium working properly?

$ microk8s.cilium status
...
$ microk8s.kubectl get pods -n kube-system
...

Cannot push Cilium image via make microk8s

$ make microk8s
...
  DEPLOY image to microk8s (localhost:32000/cilium/cilium:local)
docker tag cilium/cilium-dev:local localhost:32000/cilium/cilium:local
docker push localhost:32000/cilium/cilium:local
The push refers to repository [localhost:32000/cilium/cilium]
Get http://localhost:32000/v2/: dial tcp 127.0.0.1:32000: connect: connection refused

This suggests that the registry is not available locally. Ensure that it has been enabled via:

microk8s.enable registry

Then ensure that the registry pods are ready:

$ microk8s.kubectl get pods -n container-registry
NAMESPACE            NAME                                    READY   STATUS              RESTARTS   AGE
container-registry   registry-d7d7c8bc9-cwjgt                1/1     Ready               0          17m

If the pods aren't ready, investigate further with kubectl describe pod .... Ensure that cilium is enabled.

Cilium pods cannot be deployed due to cluster policy

This error occurs when deploying Cilium:

Forbidden: disallowed by cluster policy microk8s

Resolution: Ensure that the microk8s kube-apiserver is configured with --allow-privileged in /var/snap/microk8s/current/args/kube-apiserver and microk8s has been restarted with this option configured (microk8s.stop; microk8s.start).

Cilium EOF when attempting to reach docker

Ensure that the docker socket has been updated in the Cilium-DS YAML:

https://gist.github.com/joestringer/60a5f53d59e57274ed4c2a1736a7b101#set-up-cilium

Restart microk8s

In case there is a change in host IP, you can restart kubernetes API server the following way in order to propagte the new IP to all kubernetes cluster members:

# microk8s.stop
# microk8s.start

Problems pushing to docker registry

The following error may occur when IP connectivity to the registry is not available:

# docker push localhost:32000/cilium/cilium:my-image
The push refers to repository [localhost:32000/cilium/cilium]
Get http://localhost:32000/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

This may occur for multiple reasons:

  • Your exernal IP has changed (for instance, your laptop was suspended and restored on a new network)
  • Your localhost attempts to connect to the registry via IPv6
    • The docker daemon is not listening on IPv4, so if the docker destination of localhost:32000 resolves to IPv6 this may cause timeouts. To overcome this, use 127.0.0.1 instead of localhost, or remove the IPv6 (::1) host alias from /etc/hosts.

Misc

Some quick-links for general troubleshooting:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment