Cilium ingress controller with cert-manager

README.md

Tutorial: Cilium ingress controller with cert-manager

👉 This tutorial is also visible in the Cilium Service Mesh official documentation here.

With this tutorial, you will install Cilium Service Mesh on Kind with TLS with certificates created by cert-manager. It is inspired by the TLS example on the Cilium website.

This was written on 18 Feb 2022 during the beta of the Cilium Service Mesh. A lot probably changed since then.

Prerequisites:

• helm v3.7 and above,
• kind v0.11.1 or above,
• kubectl,
• cilium CLI v0.10.2 and above.

kind create cluster --config /dev/stdin <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
EOF

cilium install --version -service-mesh:v1.11.0-beta.1 --config enable-envoy-config=true --kube-proxy-replacement=probe
cilium hubble enable

# https://kind.sigs.k8s.io/docs/user/loadbalancer/
helm repo add --force-update metallb https://metallb.github.io/metallb
NET=$(docker network inspect kind | jq '.[].IPAM.Config[0] | select(.Gateway) | .Subnet' -r | sed 's|/16||') \
    && helm upgrade --install metallb metallb/metallb --version 0.12.1 --values /dev/stdin --create-namespace -n metallb-system <<EOF
configInline:
  address-pools:
    - {name: default, protocol: layer2, addresses: ["${NET/%.0/.200}-${NET/%.0/.250}"]}
EOF

Now, let us install cert-manager:

helm repo add --force-update jetstack https://charts.jetstack.io
helm upgrade --install cert-manager jetstack/cert-manager --version v1.7.1 --namespace cert-manager --set installCRDs=true --create-namespace
kubectl apply -f- <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: self-signed
  namespace: default
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca
  namespace: default
spec:
  isCA: true
  privateKey:
    algorithm: ECDSA
    size: 256
  secretName: ca
  commonName: ca
  issuerRef:
    name: self-signed
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ca-issuer
  namespace: default
spec:
  ca:
    secretName: ca
EOF

Finally, let us install a demo application, and configure your DNS to resolve the domains ookinfo.cilium.rocks and hipstershop.cilium.rocks:

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.11/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium-service-mesh-beta/e5c1f5b/kubernetes-ingress/tls-ingress.yaml
kubectl annotate ingress tls-ingress cert-manager.io/issuer=ca-issuer
sudo perl -ni -e 'print if !/\.cilium\.rocks$/d' /etc/hosts; sudo tee -a /etc/hosts <<<"$(kubectl get svc/cilium-ingress-tls-ingress -o=jsonpath='{.status.loadBalancer.ingress[0].ip}' | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" || printf "\e[31mError: loadbalancer ip not ready yet, try again later\e[0m\n" >&2) bookinfo.cilium.rocks hipstershop.cilium.rocks"
kubectl wait --for=condition=Ready pods -n default --all

Now, let's test it:

curl --cacert <(kubectl get secret ca -o="jsonpath={.data.ca\.crt}" | base64 --decode) https://bookinfo.cilium.rocks/details/1

You should see:

{"id":1,"author":"William Shakespeare","year":1595,"type":"paperback","pages":200,"publisher":"PublisherA","language":"English","ISBN-10":"1234567890","ISBN-13":"123-1234567890"}

SSL routines:ssl3_get_record:wrong version number

The Cilium Service Mesh ingress controller seems to fallback on serving over HTTP when the demo-cert Kubernetes Secret hasn't been created yet by cert-manager. There might be an option for "strict HTTPS", I haven't found it yet.

One issue the Cilium Service Mesh ingress controller currently has is that it won't dynamically "reload" the ingress configuration when the demo-cert becomes available. When demo-cert becomes availabe, I noticed that you need to recreate or update the Ingress:

kubectl delete -f https://raw.githubusercontent.com/cilium/cilium-service-mesh-beta/e5c1f5b/kubernetes-ingress/tls-ingress.yaml
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium-service-mesh-beta/e5c1f5b/kubernetes-ingress/tls-ingress.yaml

Details: when the demo-cert hasn't been created yet, you will see the message:

curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

That happens because curl (or rather, openssl) has successfully handshaked the TCP connection, and sent its Client Hello. At this point, it is expecting a Server Hello. With Wireshark open, we run:

$ curl https://bookinfo.cilium.rocks
CONNECTED(00000003)
139744837105152:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:332:

The Wireshark/tcpdump command:

docker run --rm --net=host --privileged itsthenetwork/alpine-tcpdump:latest host 172.18.0.200 -i $(ip --json route get 172.18.0.2 | jq -r '.[].dev') -U -w - | wireshark -k -y RAW -i -

We can see that the Client Hello was sent, an ACK is sent by the server, and the server sends HTTP/1.1 400 Bad Request:

curl wasn't expecting HTTP/1.1; instead, it was expecting a payload of type Server Hello.