Whereabouts alternative version installation + test assets

README.md

Alternative Whereabouts version installation on OCP

This is for installing a fix for Whereabouts to avoid a race condition which could cause duplicate IPs when used at scale.

NOTE: If you also have the CNO running, it's possible that if the CNO upgrades Whereabouts -- it may clobber this installation. In which case, you must run these steps (or a subset of them) over again (particular the creation of the daemonset-install.yaml.

Installation

Firstly, create the addition RBAC, using the included rbac.yaml file with:

oc create -f rbac.yaml

Create the daemonset-install.yaml text file as shown in this gist, and create with:

oc create -f daemonset-install.yaml

Verify it by looking for pods named whereabouts-* such as:

oc get pods -n openshift-multus

Verification

Create the net-attach-def yaml file as provided, net-attach-def.yaml.

NOTE you must change the value of the master parameter to match a real network interface name on all machines.

Then create with oc create -f net-attach-def.yaml

Create the replica set yaml, replica-set.yaml.

Edit the replica set yaml number of replicas to match your desired number of replicas, e.g. change replicas: 3 to say replicas: 200

NOTE You must create the net-attach-def, and the replica set in the same namespace.

You may check for the number of pods that have gone into a running state with:

watch -n1 "oc get pods -o wide | grep Running | wc -l"

Checking for duplicate IPs

This command executes ip a in each pod named like whereabouts-test-*, and then sorts those IP addresses

NOTE Change inet 198 to match part of the IP address in the range parameter in the net-attach-def as created earlier.

oc get pods | grep whereabouts-test | awk '{print $1}' | xargs -I {} oc exec -t {} -- ip a | grep "inet 198" | awk '{print $2}' | sort | uniq | wc -l

This should output the same number as the number of replicas

If the number is less than the number of replicas, there is a duplicated IP address.

daemonset-install.yaml

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: whereabouts
  namespace: openshift-multus
  labels:
    tier: node
    app: whereabouts
spec:
  selector:
    matchLabels:
      name: whereabouts
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        tier: node
        app: whereabouts
        name: whereabouts
    spec:
      hostNetwork: true      
      serviceAccountName: multus
      tolerations:
      - operator: Exists
      containers:
      - name: whereabouts
        image: quay.io/dougbtv/whereabouts:leaderelection
        env:
        - name: SLEEP
          value: "true"
        - name: CNI_CONF_DIR
          value: "/host/etc/kubernetes/cni/net.d"
        - name: CNI_BIN_DIR
          value: "/host/var/lib/cni/bin"
        - name: WHEREABOUTS_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        volumeMounts:
        - name: cnibin
          mountPath: /host/var/lib/cni/bin
        - name: cni-net-dir
          mountPath: /host/etc/kubernetes/cni/net.d
      volumes:
        - name: cnibin
          hostPath:
            path: /var/lib/cni/bin
        - name: cni-net-dir
          hostPath:
            path: /etc/kubernetes/cni/net.d

net-attach-def.yaml

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "ens4",
      "mode": "bridge",
      "ipam": {
        "type": "whereabouts",
        "range": "198.18.0.0/15"
      }
    }'

rbac.yaml

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: multus-whereabouts-lease
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: whereabouts-cni-lease
subjects:
- kind: ServiceAccount
  name: multus
  namespace: openshift-multus

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: whereabouts-cni-lease
rules:
- apiGroups: ["coordination.k8s.io"]
  resources: 
  - leases
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete

replica-set.yaml

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: whereabouts-test
  labels:
    app: whereabouts-test
    tier: whereabouts-test
spec:
  # modify replicas according to your case
  replicas: 3
  selector:
    matchLabels:
      tier: whereabouts-test
  template:
    metadata:
      labels:
        tier: whereabouts-test
      annotations:
        k8s.v1.cni.cncf.io/networks: macvlan-conf
    spec:
      containers:
      - name: samplepod
        command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
        image: quay.io/dougbtv/alpine:latest