install-cilium-olm.md

NOTE: Do make sure Cilium ports are open before proceeding (see https://docs.cilium.io/en/v1.10/gettingstarted/k8s-install-openshift-okd/)

Create the namnespace:

kubectl apply -f - << EOF
---
apiVersion: v1
kind: Namespace
metadata:
  name: cilium
  annotations:
    # node selector is required to make cilium-operator run on control plane nodes
    openshift.io/node-selector: ""
  labels:
    name: cilium
    # enable cluster logging for Cilium namespace
    openshift.io/cluster-logging: "true"
    # enable cluster monitoring for Cilium namespace
    openshift.io/cluster-monitoring: "true"
EOF

Create custom SCCs with least privilages needed to run Cilium:

kubectl apply -f - << EOF
---
apiVersion: security.openshift.io/v1

kind: SecurityContextConstraints
metadata:
  name: cilium

users:
- system:serviceaccount:cilium:cilium

allowPrivilegeEscalation: true
allowPrivilegedContainer: true

defaultAddCapabilities: null
requiredDropCapabilities: null

allowedCapabilities:
- NET_ADMIN
- SYS_MODULE

allowedUnsafeSysctls: null

allowHostNetwork: true

volumes:  
- hostPath
- configMap
- secret

priority: null

readOnlyRootFilesystem: false
allowHostDirVolumePlugin: true

allowHostIPC: false
allowHostPID: false
allowHostPorts: false

runAsUser:
  type: RunAsAny

seLinuxContext:
  type: RunAsAny

seccompProfiles: null

supplementalGroups:
  type: RunAsAny

fsGroup:
  type: RunAsAny
---
apiVersion: security.openshift.io/v1

kind: SecurityContextConstraints
metadata:
  name: cilium-operator

users:
- system:serviceaccount:cilium:cilium-operator

allowPrivilegeEscalation: false
allowPrivilegedContainer: false

defaultAddCapabilities: null
requiredDropCapabilities: null

allowedCapabilities: null

allowedUnsafeSysctls: null

allowHostNetwork: true

volumes:  
- configMap
- secret

priority: null

readOnlyRootFilesystem: false
allowHostDirVolumePlugin: false

allowHostIPC: false
allowHostPID: false
allowHostPorts: false

runAsUser:
  type: RunAsAny

seLinuxContext:
  type: RunAsAny

seccompProfiles: null

supplementalGroups:
  type: RunAsAny

fsGroup:
  type: RunAsAny
EOF

Create OLM resources to install from RedHat catalogue:

kubectl apply -f - << EOF
---
apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: cilium
  namespace: cilium
spec:
  targetNamespaces:
  - cilium
--- 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cilium
  namespace: cilium
spec:
  channel: stable
  installPlanApproval: Automatic
  name: cilium
  source: certified-operators
  sourceNamespace: openshift-marketplace
  startingCSV: cilium.v1.10.4-x5bfd7b3
EOF

Wait for the operator to get installed:

until kubectl wait deployment --for="condition=Available" --namespace="cilium" cilium-olm ; do sleep 0.5 ; done
kubectl wait crd --for="condition=Established" ciliumconfigs.cilium.io

Create CiliumConfig (this assumes default IPI installation):

kubectl apply -f - << EOF
---
apiVersion: cilium.io/v1alpha1
kind: CiliumConfig
metadata:
  name: cilium
  namespace: cilium
spec:
  kubeProxyReplacement: "probe"
  ipam:
    mode: "cluster-pool"
    operator:
      clusterPoolIPv4PodCIDR: "10.128.0.0/14"
      clusterPoolIPv4MaskSize: "23"
  nativeRoutingCIDR: "10.128.0.0/14"
  cni:
    binPath: "/var/lib/cni/bin"
    confPath: "/var/run/multus/cni/net.d"
  prometheus:
    serviceMonitor: {enabled: false}
  hubble:
    tls: {enabled: false}
  # UBI-specific settings
  image:
    repository: registry.connect.redhat.com/isovalent/cilium
    tag: bde97f4465c67b00a4d58ec20463eec5ba307eb1-v1.10.4-ubirh8
    digest: sha256:631082dcad78fac902e948aefecee3bf05ff61ababfa1a64c086747fc2b2cb1c
  operator:
    image:
      repository: registry.connect.redhat.com/isovalent/operator
      tag: d47e3df3d1567eb1d694189db5390c54437c5f0e-v1.10.4-ubirh8
      genericDigest: sha256:b0e01e8499b4be7620ff45f9e4af8b8f158513de295aaac8bc0a725b7266dc25
  extraHostPathMounts:
  - name: host-iptables-lib64
    mountPath: /host/iptables/lib64
    hostPath: /lib64
    hostPathType: Directory
    readOnly: true
    mountPropagation: HostToContainer
  - name: host-iptables-usr-lib64-xtables
    mountPath: /host/iptables/usr/lib64/xtables
    hostPath: /usr/lib64/xtables
    hostPathType: Directory
    readOnly: true
  - name: host-iptables-usr-sbin
    mountPath: /host/iptables/usr/sbin
    hostPath: /usr/sbin
    hostPathType: Directory
    readOnly: true
    mountPropagation: HostToContainer
  - name: host-iptables-etc-alternatives
    mountPath: /host/iptables/etc/alternatives
    hostPath: /etc/alternatives
    hostPathType: Directory
    readOnly: true
    mountPropagation: HostToContainer
  - name: host-iptables-run-xtables-lock
    mountPath: /host/iptables/run/xtables.lock
    hostPath: /run/xtables.lock
    hostPathType: FileOrCreate
    readOnly: false
    mountPropagation: HostToContainer
  - name: host-iptables-proc
    mountPath: /host/iptables/proc
    hostPath: /proc
    hostPathType: Directory
    readOnly: true
    mountPropagation: HostToContainer
EOF

Wait for Cilium pods:

until kubectl wait deployment --for="condition=Available" --namespace="cilium" cilium-operator ; do sleep 0.5 ; done
until test "$(kubectl get daemonset --namespace=cilium cilium --output="jsonpath={.status.numberReady}")" -eq "$(kubectl get nodes --output="jsonpath={range .items[*]}{.metadata.name}{\"\n\"}{end}" | wc -l)" ; do sleep 0.5 ; done

Initialise network migration (based on 4.6 docs):

kubectl annotate network.operator cluster 'networkoperator.openshift.io/network-migration'=""
kubectl patch machineconfigpool worker master --type='merge' --patch  '{ "spec": { "paused": true } }'
kubectl patch network.config cluster --type='merge' --patch '{ "spec": { "networkType": "Cilium" } }'

Wait for Multus rollout to complete:

# kubectl rollout restart --namespace=openshift-multus daemonset multus
kubectl rollout status --namespace=openshift-multus daemonset multus

Reboot every node:

for i in $(kubectl get nodes  -o jsonpath='{.items[*].metadata.name}') ; do oc debug "node/${i}" -- chroot /host shutdown -r -t 3 ; done

Finish network migration:

kubectl patch machineconfigpool worker master --type='merge' --patch  '{ "spec": { "paused": false } }'
kubectl annotate network.operator cluster 'networkoperator.openshift.io/network-migration-'

Delete openshift-sdn namespace:

kubectl delete ns openshift-sdn

Patch status.networkType in network.config/cluster:

kubectl get network.config cluster --output=json | jq '.status.networkType="Cilium"' | kubectl apply -f -