add-ingresscontroller

This gist demonstrates a complete workflow for adding a 2nd ingress controller (ic) to an OpenShift 4.x IPI baremetal platform. 
It is in no way complete, lacking testing of many options, but is more complete than any other source I know of.
The main drawback of this deployment type is that you have 3 controllers; so if the default ingresscontroller has a 
replicasetof 2 then the additional ic must have replicaset=1.  This is because, by default, the routers for the two 
ic's listen on the same ports.  

Helpful KCS: https://access.redhat.com/solutions/6017531 

A) Create an additional ingress controller

Set the nodeSelector to run where the default ic is NOT running:

[root@openshift-jumpserver-0 ingresscontrollers]# cat my-custom-ic.yaml 
apiVersion: v1
items:
- apiVersion: operator.openshift.io/v1
  kind: IngressController
  metadata:
    name: custom-ic
    namespace: openshift-ingress-operator
  spec:
    replicas: 1
    domain: my-custom-ic.ipi-cluster.example.com
    nodePlacement:
      nodeSelector:
        matchLabels:
          ingresscontroller: "custom-ic"
    namespaceSelector:
      matchLabels:
        ic: custom-ic
    endpointPublishingStrategy:
      type: HostNetwork
      #type: LoadBalancerService
  status: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Label the node that does not have a default IC router running on it
oc label node openshift-worker-2 ingresscontroller=custom-ic 
oc create -f my-custom-ic.yaml -n openshift-ingress-operator 

oc get pods -n openshift-ingress -o wide
NAME                                 READY   STATUS    RESTARTS   AGE   IP                NODE                 NOMINATED NODE   READINESS GATES
router-default-5bdfc578d6-b5rzr      1/1     Running   0          26m   X.Y.Z.220   openshift-worker-0   <none>           <none>
router-default-5bdfc578d6-sxcq9      1/1     Running   0          26m   X.Y.Z.222   openshift-worker-2   <none>           <none>
custom-ic-router-5c9f964d9d-qz782     1/1     Running   0          18m   X.Y.Z.221   openshift-worker-1   <none>           <none>

oc new-project custom-ic 
oc label namespace custom-ic ic=custom-ic
oc new-app --image "docker.io/openshift/hello-openshift"

B) Create the wildcard DNS entry for the IC

The short cut:
Must have a DNS entry for the route. ADD DNS to IPI baremetal deployment to /etc/hosts:
root@openshift-jumpserver-0 ingresscontrollers]# grep hello-openshift /etc/hosts
X.Y.Z.221 hello-openshift.my-custom-ic.ipi-cluster.example.com <<<---IP of router for this IC
[root@openshift-jumpserver-0 ingresscontrollers]# ping hello-openshift.apps-ic2.ipi-cluster.example.com
PING hello-openshift.my-custom-ic.ipi-cluster.example.com (X.Y.Z.221) 56(84) bytes of data.
64 bytes from hello-openshift.my-custom-ic.ipi-cluster.example.com (X.Y.Z.221): icmp_seq=1 ttl=64 time=0.759 ms

The right way by adding a wildcard dns:
*.my-custom-ic.ipi-cluster.example.com   IN   A X.Y.Z.22

C) create the route

oc expose svc/hello-openshift --hostname=hello-openshift.my-custom-ic.ipi-cluster.example.com

[root@openshift-jumpserver-0 ingresscontrollers]# curl http://hello-openshift.my-custom-ic.ipi-cluster.example.com
Hello OpenShift! 

NOTE: at this point we have two potential routes  The route tries to tell us that:
[root@openshift-jumpserver-0 new]# oc describe route/hello-openshift | grep "Requested Host" -A 2
Requested Host:		hello-openshift.my-custom-ic.ipi-cluster.example.com
  exposed on router custom-ic-router (host router-custom-ic-router.my-custom-ic.ipi-cluster.example.com)
  exposed on router default (host router-default.apps.ipi-cluster.example.com) 

Instead, delete that route and create a route that will be served onlyl from this IC

To create a route that has only one IC, do the following:
[root@openshift-jumpserver-0 ingresscontrollers]# cat my-custom-ic-route.yaml 
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  labels:
          ic: custom-ic
  name: hello-openshift
  namespace: custom-ic
spec:
  host: hello-openshift.my-custom-ic.ipi-cluster.example.com
  port:
    targetPort: 8080-tcp
  to:
    kind: Service
    name: hello-openshift

oc create -f ./my-custom-ic-route.yaml -n custom-ic

Now there is only one route:
[root@openshift-jumpserver-0 ingresscontrollers]# oc get route
NAME              HOST/PORT                                              PATH   SERVICES          PORT       TERMINATION   WILDCARD
hello-openshift   hello-openshift.my-custom-ic.ipi-cluster.example.com          hello-openshift   8080-tcp                 None

D) Check the IC logs to confirm that routes are being service from the indended ic's

To find out which IC your traffic is going through, turn on ic destination logging (or confirm with curl --resolve method in https://access.redhat.com/solutions/5097511 ):
oc patch -n openshift-ingress-operator ingresscontrollers/custom-ic  --patch '{"spec":{"logging":{"access":{"destination":{"type":"Container"}}}}}' --type merge

Now there is a “logs” container on the custom-ic-router pod:
oc get pods/router-custom-ic-6fb58d7d67-dztdj -n openshift-ingress  -o json | jq '.spec.containers[].name' 
"router"
"logs"

To look at logs for custom ic:
oc logs router-custom-ic-6fb58d7d67-dztdj -c logs  -n openshift-ingress --follow

If you curl the route then you see this in the custom-ic log:
2023-05-11T20:17:41.561350+00:00 openshift-worker-2 openshift-worker-2 haproxy[48]: X.Y.Z.1:37636 [11/May/2023:20:17:41.560] public be_http:custom-ic:hello-openshift/pod:hello-openshift-656ffd458b-wjqwr:hello-openshift:8080-tcp:172.27.0.145:8080 0/0/0/1/1 200 142 - - --NI 1/1/0/0/0 0/0 "GET / HTTP/1.1"

If you watch the logs of the default ic at the same time you don’t see this destination:
oc -n openshift-ingress logs deployment.apps/router-default -c logs

To look at logs for ingress operator (informational, I did not see a real reason to do this):
oc logs -f --namespace=openshift-ingress-operator deployments/ingress-operator

Modify the default IC to exclude the new route

So I don’t know why you would need to exclude this namespace from the default IC. But to do it follow the instructions below.  BUT, this namespaceSelector was already in place so I don’t know if it got done automatically or if my ‘oc edit’ commands took that OCP said were not valid:

cat <<'EOF' > operator-default-patch.yaml
spec:
  namespaceSelector:
    matchExpressions:
    - key: ic
      operator: NotIn
      values:
      - custom-ic 
EOF

oc patch -n openshift-ingress-operator ingresscontroller default --type="merge" -p "$(cat operator-default-patch.yaml)"
OR turn this command into one line JSON:
[root@openshift-jumpserver-0 ~]# oc get ingresscontroller/default  -n openshift-ingress-operator -o json | jq '.spec.namespaceSelector'
{"matchExpressions": [{"key": "ic","operator": "NotIn","values": ["custom-ic"]}]}

E) Createthe VIP for the new IC

Note: the nodeSelector must match the node where the additional ic is running: 

[root@openshift-jumpserver-0 ingresscontrollers]# cat my-custom-ic-vip.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ipfailover-keepalived 
  labels:
    ipfailover: custom-ic-vip
spec:
  strategy:
    type: Recreate
  replicas: 2
  selector:
    matchLabels:
      ipfailover: custom-ic-vip
  template:
    metadata:
      labels:
        ipfailover: custom-ic-vip
    spec:
      serviceAccountName: ipfailover
      privileged: true
      hostNetwork: true
      nodeSelector:
        ingresscontroller: "custom-ic"
      containers:
      - name: openshift-ipfailover
        image: quay.io/openshift/origin-keepalived-ipfailover
        ports:
        - containerPort: 63000
          hostPort: 63000
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
        volumeMounts:
        - name: lib-modules
          mountPath: /lib/modules
          readOnly: true
        - name: host-slash
          mountPath: /host
          readOnly: true
          mountPropagation: HostToContainer
        - name: etc-sysconfig
          mountPath: /etc/sysconfig
          readOnly: true
        - name: config-volume
          mountPath: /etc/keepalive
        env:
        - name: OPENSHIFT_HA_CONFIG_NAME
          value: "ipfailover"
        - name: OPENSHIFT_HA_VIRTUAL_IPS 
          value: "X.Y.Z.22" #changed this
        - name: OPENSHIFT_HA_VIP_GROUPS 
          value: "1" #changed this
        - name: OPENSHIFT_HA_NETWORK_INTERFACE 
          value: "br-ex" #The host interface to assign the VIPs; changed this
        - name: OPENSHIFT_HA_MONITOR_PORT 
          value: "0" #changed this
        - name: OPENSHIFT_HA_VRRP_ID_OFFSET 
          value: "1" #changed this
        - name: OPENSHIFT_HA_REPLICA_COUNT 
          value: "1" #Must match the number of replicas in the deployment; changed this
        - name: OPENSHIFT_HA_USE_UNICAST
          value: "true" #changed this
        - name: OPENSHIFT_HA_UNICAST_PEERS
          value: "X.Y.Z.220,X.Y.Z.221,X.Y.Z.222"
        - name: OPENSHIFT_HA_IPTABLES_CHAIN 
          value: "INPUT"
        #- name: OPENSHIFT_HA_NOTIFY_SCRIPT 
        #  value: /etc/keepalive/mynotifyscript.sh
        - name: OPENSHIFT_HA_CHECK_SCRIPT 
          value: "/etc/keepalive/mycheckscript.sh"
        - name: OPENSHIFT_HA_PREEMPTION 
          value: "preempt_delay 300"
        - name: OPENSHIFT_HA_CHECK_INTERVAL 
          value: "5"
        livenessProbe:
          initialDelaySeconds: 10
          exec:
            command:
            - pgrep
            - keepalived
      volumes:
      - name: lib-modules
        hostPath:
          path: /lib/modules
      - name: host-slash
        hostPath:
          path: /
      - name: etc-sysconfig
        hostPath:
          path: /etc/sysconfig
      # config-volume contains the check script
      # created with `oc create configmap keepalived-checkscript --from-file=mycheckscript.sh`
      - configMap:
          defaultMode: 0755
          name: keepalived-checkscript
        name: config-volume
      imagePullSecrets:
        - name: openshift-pull-secret 

Note: I am completely ignoring the check script:
echo “echo no-op” >./mycheckscript.sh
oc create configmap keepalived-checkscript --from-file=mycheckscript.sh
oc create -f ./my-custom-ic-vip.yaml -n ipfailover

Check that pods in ipfailover project were created, then check to see where the VIP was provisioned - should be the node designated by the nodeSelector:
oc get nodes -o yaml | grep -e host-addresses -e ' name:'

F) confirm the results

curl  hello-openshift.my-custom-ic.ipi-cluster.example.com 
Hello OpenShift!

G) The following demonstrates how to scale down the default ingresscontroller to 1 node so the custom ingress controller can run on 2:

Label one of the node currently running the default IC:
oc label node openshift-worker-2 ingresscontroller=default-ic

Patch the default ic to have one replica and select only a labeled node:
cat <<'EOF' > operator-default-patch2.yaml
spec:
  nodePlacement:
    nodeSelector:
      matchLabels:
        ingresscontroller: default-ic
  replicas: 1
EOF

oc patch -n openshift-ingress-operator ingresscontroller default --type="merge" -p "$(cat operator-default-patch2.yaml)"

Label the extra node for running the custom IC:
oc label node openshift-worker-0 ingresscontroller=custom-ic

Scale up the custom IC:
oc scale ingresscontroller/custom-ic -n openshift-ingress-operator --replicas=2 

Scale up the failover IP:
oc scale deployments/ipfailover-keepalived -n ipfailover --r

oc get pods -n openshift-ingress -o wide
NAME                                READY   STATUS    RESTARTS   AGE    IP                NODE                 NOMINATED NODE   READINESS GATES
router-custom-ic-859fbb5959-hth85   2/2     Running   0          30m    X.Y.Z.220   openshift-worker-0   <none>           <none>
router-custom-ic-859fbb5959-jw5mz   2/2     Running   0          24h    X.Y.Z.221   openshift-worker-1   <none>           <none>
router-default-df9486bf5-c9dx4      2/2     Running   0          118m   X.Y.Z.222   openshift-worker-2   <none>           <none>

You should be able to reboot either node running the custom IC (worker-0,worker-1) and the failover IP and the active router 
should be active on the other.