Whereabouts Demo

README.md

A demo of the Whereabouts plugin.

Pre-reqs.

• Install Multus CNI.
• Install etcd operator & etcd.

Demo

Clone Whereabouts and install it...

git clone https://github.com/dougbtv/whereabouts.git && cd whereabouts
kubectl apply -f ./doc/daemonset-install.yaml

Label our hosts so we can have pods appear on specific nodes...

kubectl label nodes kube-nonetwork-node-1 side=left
kubectl label nodes kube-nonetwork-node-2 side=right
kubectl get nodes --show-labels

Get the etcd IP address (or service name if you can resolve it from the host [recommended])

export ETCDHOST=$(kubectl get svc | grep cluster-client | awk '{print $3}')

Create the CR with a whereabouts IPAM section.

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
      "cniVersion": "0.3.0",
      "name": "whereaboutsexample",
      "type": "macvlan",
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "whereabouts",
        "etcd_host": "$ETCDHOST:2379",
        "range": "192.168.2.225/28",
        "exclude": [
           "192.168.2.229/30",
           "192.168.2.236/32"
        ],
        "log_file" : "/tmp/whereabouts.log",
        "log_level" : "debug",
        "gateway": "192.168.2.1"
      }
    }'
EOF

Create a couple pod specs.

$ cat samplepod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
  containers:
  - name: samplepod
    command: ["/bin/bash", "-c", "sleep 2000000000000"]
    image: centos/tools
  nodeSelector:
    side: left

$ cat samplepod2.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: samplepod2
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
  containers:
  - name: samplepod2
    command: ["/bin/bash", "-c", "sleep 2000000000000"]
    image: centos/tools
  nodeSelector:
    side: right

Now go ahead and create the first one...

$ kubectl create -f samplepod.yaml

Note that it's on the first worker node, because of the node selector...

kubectl get pod samplepod -o wide

Exec ip a in that pod...

kubectl exec -it samplepod -- ip a

Note that it allocates an IP address in the range we have in the configuration.

Let's see that address as allocated in etcd.

kubectl exec -it $(kubectl get pods | grep "etcd-cluster" | head -n1 | awk '{print $1}') -- sh -c 'ETCDCTL_API=3 etcdctl get /192.168.2.225/28'

Check out that it has a few components:

/192.168.2.225/28

192.168.2.224 78e5db4717b3aa44a69835dad7ffe676532e9977b00976660c21d3c149bf9b42

The top line is the range for which the allocations are stored.

The bottom line is two parts:

1. The IP address allocated
2. The pod sandbox ID.

Now, let's start another pod...

kubectl exec -it samplepod2 -- ip a

You can see it's on the second node, because of the node selector...

kubectl get pod samplepod2 -o wide

Query etcd again, with the same command as above, we now have two IPs allocated...

[centos@kube-nonetwork-master ~]$ kubectl exec -it $(kubectl get pods | grep "etcd-cluster" | head -n1 | awk '{print $1}') -- sh -c 'ETCDCTL_API=3 etcdctl get /192.168.2.225/28'
/192.168.2.225/28

192.168.2.224 78e5db4717b3aa44a69835dad7ffe676532e9977b00976660c21d3c149bf9b42
192.168.2.225 b3ff7f5599d18624a2270382a000ea678e0df5419ebb5c4a88e7826fc8119f36

And now we can delete the first pod, and see that it deallocates it...

[centos@kube-nonetwork-master ~]$ kubectl delete pod samplepod
pod "samplepod" deleted
[centos@kube-nonetwork-master ~]$ kubectl exec -it $(kubectl get pods | grep "etcd-cluster" | head -n1 | awk '{print $1}') -- sh -c 'ETCDCTL_API=3 etcdctl get /192.168.2.225/28'
/192.168.2.225/28

192.168.2.225 b3ff7f5599d18624a2270382a000ea678e0df5419ebb5c4a88e7826fc8119f36