This details a reference deployment of Istio w/ Multus CNI to demonstrate a problem where annotations are being clobbered by the Istio webhook. It also provides a patch and workflow for a possible fix.
This article first demonstrates how to reproduce the article, then proposes a patch, and demonstrates a way to build and deploy Istio with the modified code.
NOTE: Ignore the 1.5.1 through the install, I replicate it with latest (Nov 2021), and provide further steps following the rest of the installation.
- Kubernetes (I used 1.18.0 and later 1.22.3)
- Some pod-to-pod CNI plugin installed (I used flannel)
- Multus CNI installed using the quickstart guide.
cat <<EOF | kubectl apply -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: istio-cni
spec:
config: ''
EOF
This enables Multus to search for the on-disk configuration that contains a JSON CNI config with the name field set to istio-cni.
Download Istio per these installation instructions.
curl -L https://istio.io/downloadIstio | sh -
cd istio-1.5.1
When going to make the installation, we use the parameters for OpenShift 4.2 from the istio-cni installation instructions.
istioctl manifest apply \
--set profile=demo \
--set components.cni.enabled=true \
--set components.cni.namespace=kube-system \
--set values.cni.cniBinDir=/opt/cni/bin \
--set values.cni.cniConfDir=/etc/cni/multus/net.d \
--set values.cni.chained=false \
--set values.cni.cniConfFileName="istio-cni.conf" \
--set values.sidecarInjectorWebhook.injectedAnnotations."k8s\.v1\.cni\.cncf\.io/networks"=istio-cni
Short description of what I believe the parameters are accomplishing:
| Parameter | Assumed purpose |
|---|---|
--set components.cni.enabled=true |
Enable istio CNI plugin |
--set components.cni.namespace=kube-system |
Namespace where istio cni daemonset runs |
--set values.cni.cniBinDir=/opt/cni/bin |
Directory where istio-cni binary will be dropped |
--set values.cni.cniConfDir=/etc/cni/multus/net.d |
Where istio-cni.conf will be written (important) |
--set values.cni.chained=false |
Disables istio-cni as a chained CNI plugin (important) |
--set values.cni.cniConfFileName="istio-cni.conf" |
Name of configuration file to write (required but actual filename can be arbitrary) |
--set values.sidecarInjectorWebhook.injectedAnnotations [...] |
Annotation that's added to pods that will use Istio. |
This results in files being created for Istio configuration, especially landing in /etc/cni/multus/net.d/
[centos@kube-singlehost-master istio-1.5.1]$ ls /etc/cni/multus/net.d/
istio-cni.conf ZZZ-istio-cni-kubeconfig
[centos@kube-singlehost-master istio-1.5.1]$ cat /etc/cni/multus/net.d/istio-cni.conf
{
"cniVersion": "0.3.1",
"name": "istio-cni",
"type": "istio-cni",
"log_level": "info",
"kubernetes": {
"kubeconfig": "/etc/cni/multus/net.d/ZZZ-istio-cni-kubeconfig",
"cni_bin_dir": "/opt/cni/bin",
"exclude_namespaces": [ "istio-system" ]
}
}
Deploying the sample application...
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
Once the pods come up, use the early validation as suggested in the install docs...
kubectl exec -it $(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}') -c ratings -- curl productpage:9080/productpage | grep -o "<title>.*</title>"
(It should say "simple bookstore app")
Latest attempt used Istio 1.11.4
I found that I followed the same steps, but! I needed to enable istio on a given namespace...
Run istioctl analyze to see if there's a problem. And then I ran:
kubectl label namespace default istio-injection=enabled
Then, I could reproduce the issue.
Create a custom resource for Multus (this uses the example from the Multus CNI quickstart guide)
cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-conf
spec:
config: '{
"cniVersion": "0.3.0",
"type": "macvlan",
"master": "eth0",
"mode": "bridge",
"ipam": {
"type": "host-local",
"subnet": "192.168.1.0/24",
"rangeStart": "192.168.1.200",
"rangeEnd": "192.168.1.216",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"gateway": "192.168.1.1"
}
}'
EOF
Then, when deploying a pod, you reference both istio-cni and macvlan-conf in the k8s.v1.cni.cncf.io/networks annotation.
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: samplepod
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf,istio-cni
spec:
containers:
- name: samplepod
command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
image: alpine
EOF
However, the result is unsatisfactory, the annotation is clobbered by Istio:
[centos@kube-singlehost-master istio-1.5.1]$ kubectl describe pod samplepod | grep "/networks:"
k8s.v1.cni.cncf.io/networks: istio-cni
Therefore macvlan-conf is never executed.
The offending lines in the istio webhook that creates the annotation can be found here. It may be as simple as checking to see if the value is set, and not overwriting it there.
I got a good amount of the workflow for making/running it from the Istio using-the-code-base docs.
It seems that as of right now, the lines that cause the problem are here in webhook.go in Istio.
I currently have a work-in-progress diff.
And I have my WIP changes in a branch in my dougbtv/istio fork.
NOTE: I then actually took this patch and cherry-picked it onto the release-1.11 branch so that I could have a stable to work against, and then built/ran that.
One issue is that the code is marked as Deprecated; should be set directly in the template instead -- and I haven't dug deep enough yet to understand what that is.
Go ahead and clone this code, apply the path, however you want to get the modified code running.
So, how are we supposed to build this thing?
Well, I found this tip where we should setup some variables for us to use...
export HUB="docker.io/dougbtv"
# TAG should equal the version of the istioctl you'll use later in the process, or the one you downloaded earlier.
export TAG=1.11.4
I built the images using this make target....
make docker
And then I pushed the images with:
make docker.push
You're going to uninstall what we reproduced it with, because we'll install our custom version.
Same as install but adds a x uninstall command and a --purge flag, like so:
./bin/istioctl x uninstall --set profile=demo --set components.cni.enabled=true --set components.cni.namespace=kube-system --set values.cni.cniBinDir=/opt/cni/bin --set values.cni.cniConfDir=/etc/cni/multus/net.d --set values.cni.chained=false --set values.cni.cniConfFileName="istio-cni.conf" --set values.sidecarInjectorWebhook.injectedAnnotations."k8s\.v1\.cni\.cncf\.io/networks"=istio-cni --purge
Note that we're setting the hub value here to match how we tagged/pushed our images...
/usr/src/istio-1.11.4/bin/istioctl manifest apply \
--set profile=demo \
--set hub=docker.io/dougbtv \
--set components.cni.enabled=true \
--set components.cni.namespace=kube-system \
--set values.cni.cniBinDir=/opt/cni/bin \
--set values.cni.cniConfDir=/etc/cni/multus/net.d \
--set values.cni.chained=false \
--set values.cni.cniConfFileName="istio-cni.conf" \
--set values.sidecarInjectorWebhook.injectedAnnotations."k8s\.v1\.cni\.cncf\.io/networks"=istio-cni
Next, let's see if we can replicate it...
Just go ahead and create our Multus-annotated pod again...
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: samplepod
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf,istio-cni
spec:
containers:
- name: samplepod
command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
image: alpine
EOF
The result is what we're looking for:
[root@kube-fedoralab-master istio-1.11.4]# kubectl describe pod samplepod | grep "/networks:"
k8s.v1.cni.cncf.io/networks: macvlan-conf,istio-cni
That'll do it.
@dougbtv you mind if I give this a test? This could be a very useful upstream contribution.