Redhat Openshift provides a matrix showing what Kubernetes API version is supported with each OpenShift 4.x release: https://access.redhat.com/solutions/4870701
F5 XC uses Kubernetes API v1.23 when this document was created. This matches with OpenShift 4.10.
Instructions to deploy OpenShift using Redhat can be found in this excellent Blog from James Wilkins: "Installing OpenShift with Assisted Installer" at https://jameswilkins.co.uk/install-openshift/
It starts with using RedHat Assisted Installer (SaaS) which is located at https://console.redhat.com/openshift/assisted-installer/clusters/ :
- Create a new Assisted Cluster, pick the latest OpenShift version 4.10.x
- Create DNS entries for
api.<clustername>.yourdomain.xyzand*.apps.<clustername>.yourdomain.xyz, e.g.
$ host api.osc1.mwlabs.net
api.osc1.mwlabs.net has address 192.168.40.8
$ host gugus.apps.osc1.mwlabs.net
gugus.apps.osc1.mwlabs.net has address 192.168.40.9
- Select 'Add Host' and create the downloadable ISO after providing an ssh public key (used later to gain ssh root access to the nodes)
- Boot desired number of VM's or baremetal servers using the downloaded ISO
- Wait for the new hosts to show up in the UI with their MAC addresses as name
- Assign role and change the hostname via UI (for this setup, 3 node cluster is used)
- Complete the deployment via UI
- Download the kubeconfig from the UI and log into the cluster using the link provided in the UI
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
osc1-m0 Ready master,worker 62m v1.23.17+16bcd69 192.168.40.70 <none> Red Hat Enterprise Linux CoreOS 410.84.202305261556-0 (Ootpa) 4.18.0-305.88.1.el8_4.x86_64 cri-o://1.23.5-15.rhaos4.10.git0bbb0d9.el8
osc1-m1 Ready master,worker 97m v1.23.17+16bcd69 192.168.40.62 <none> Red Hat Enterprise Linux CoreOS 410.84.202305261556-0 (Ootpa) 4.18.0-305.88.1.el8_4.x86_64 cri-o://1.23.5-15.rhaos4.10.git0bbb0d9.el8
osc1-m2 Ready master,worker 95m v1.23.17+16bcd69 192.168.40.31 <none> Red Hat Enterprise Linux CoreOS 410.84.202305261556-0 (Ootpa) 4.18.0-305.88.1.el8_4.x86_64 cri-o://1.23.5-15.rhaos4.10.git0bbb0d9.el8
m1:osc1 mwiget$
Check pods
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
assisted-installer assisted-installer-controller-zhb9v 0/1 Completed 0 107m
openshift-apiserver-operator openshift-apiserver-operator-c9856c869-7nc7d 1/1 Running 2 (74m ago) 107m
openshift-apiserver apiserver-6c75c44444-8sscg 2/2 Running 0 53m
openshift-apiserver apiserver-6c75c44444-jpk9t 2/2 Running 0 54m
openshift-apiserver apiserver-6c75c44444-zphhc 2/2 Running 0 56m
openshift-authentication-operator authentication-operator-5fcf9987f8-fgl4z 1/1 Running 3 (76m ago) 106m
openshift-authentication oauth-openshift-7ccf7866d5-6s89k 1/1 Running 0 62m
openshift-authentication oauth-openshift-7ccf7866d5-gt7lr 1/1 Running 0 78m
openshift-authentication oauth-openshift-7ccf7866d5-xgm8z 1/1 Running 0 68m
openshift-cloud-controller-manager-operator cluster-cloud-controller-manager-operator-94944dcf5-lcjx5 2/2 Running 2 (77m ago) 106m
openshift-cloud-credential-operator cloud-credential-operator-6f7bbd4f95-vtj67 2/2 Running 1 (75m ago) 106m
openshift-cluster-machine-approver machine-approver-6db89bd58f-msslt 2/2 Running 2 (74m ago) 106m
openshift-cluster-node-tuning-operator cluster-node-tuning-operator-559fb89d8c-mss9p 1/1 Running 2 (74m ago) 106m
openshift-cluster-node-tuning-operator tuned-2fx7c 1/1 Running 0 62m
openshift-cluster-node-tuning-operator tuned-7rbsh 1/1 Running 0 92m
openshift-cluster-node-tuning-operator tuned-fcdnt 1/1 Running 0 92m
openshift-cluster-samples-operator cluster-samples-operator-59c4cf9498-hqrx5 2/2 Running 0 82m
openshift-cluster-storage-operator cluster-storage-operator-76b4459b89-jndc4 1/1 Running 2 (76m ago) 106m
openshift-cluster-storage-operator csi-snapshot-controller-78f6cd5486-ld2lv 1/1 Running 0 95m
openshift-cluster-storage-operator csi-snapshot-controller-78f6cd5486-lv7qj 1/1 Running 2 (74m ago) 95m
openshift-cluster-storage-operator csi-snapshot-controller-operator-5f88d78ff4-rmcm4 1/1 Running 1 (74m ago) 106m
openshift-cluster-storage-operator csi-snapshot-webhook-7477d9f5db-7657q 1/1 Running 0 95m
openshift-cluster-storage-operator csi-snapshot-webhook-7477d9f5db-kgd9m 1/1 Running 0 95m
openshift-cluster-version cluster-version-operator-5465f85fcd-l9tvf 1/1 Running 0 107m
openshift-config-operator openshift-config-operator-668d959bf9-sn4zl 1/1 Running 3 (75m ago) 106m
openshift-console-operator console-operator-c97d89579-9wkc4 1/1 Running 1 (74m ago) 79m
openshift-console console-669ccc4b94-x5cjd 1/1 Running 0 78m
openshift-console console-669ccc4b94-zx66l 1/1 Running 0 78m
openshift-console downloads-65c74ff5f4-fcpzl 1/1 Running 0 78m
openshift-console downloads-65c74ff5f4-h78qd 1/1 Running 0 78m
openshift-controller-manager-operator openshift-controller-manager-operator-5d4846478f-zqbd8 1/1 Running 3 (74m ago) 106m
openshift-controller-manager controller-manager-5nd4q 1/1 Running 0 82m
openshift-controller-manager controller-manager-qsvf7 1/1 Running 1 (74m ago) 82m
openshift-controller-manager controller-manager-x4wxr 1/1 Running 0 62m
openshift-dns-operator dns-operator-6cf598b679-x55ts 2/2 Running 0 107m
openshift-dns dns-default-4bkrp 2/2 Running 0 92m
openshift-dns dns-default-hgv2h 2/2 Running 0 59m
openshift-dns dns-default-zw9hq 2/2 Running 0 92m
openshift-dns node-resolver-klg2s 1/1 Running 0 92m
openshift-dns node-resolver-wnqks 1/1 Running 0 92m
openshift-dns node-resolver-x78gl 1/1 Running 0 62m
openshift-etcd-operator etcd-operator-68df98b66d-zvgrh 1/1 Running 2 (76m ago) 106m
openshift-etcd etcd-osc1-m0 5/5 Running 0 55m
openshift-etcd etcd-osc1-m1 5/5 Running 0 53m
openshift-etcd etcd-osc1-m2 5/5 Running 0 51m
openshift-etcd etcd-quorum-guard-59454fbbf7-8c9ng 1/1 Running 0 94m
openshift-etcd etcd-quorum-guard-59454fbbf7-hh6t7 1/1 Running 0 94m
openshift-etcd etcd-quorum-guard-59454fbbf7-lv5pf 1/1 Running 0 94m
openshift-etcd installer-2-osc1-m1 0/1 Completed 0 94m
openshift-etcd installer-3-osc1-m2 0/1 Error 0 91m
openshift-etcd installer-3-retry-1-osc1-m2 0/1 Completed 0 87m
openshift-etcd installer-4-osc1-m1 0/1 Completed 0 82m
openshift-etcd installer-4-osc1-m2 0/1 Completed 0 84m
openshift-etcd installer-6-osc1-m0 0/1 Completed 0 62m
openshift-etcd installer-8-osc1-m0 0/1 Completed 0 58m
openshift-etcd installer-8-osc1-m1 0/1 Completed 0 54m
openshift-etcd installer-8-osc1-m2 0/1 Completed 0 52m
openshift-image-registry cluster-image-registry-operator-765cc59ccf-tp8pk 1/1 Running 2 (74m ago) 106m
openshift-image-registry node-ca-9vv4n 1/1 Running 0 70m
openshift-image-registry node-ca-lpkmm 1/1 Running 0 62m
openshift-image-registry node-ca-pxxpc 1/1 Running 0 70m
openshift-ingress-canary ingress-canary-hggjd 1/1 Running 0 79m
openshift-ingress-canary ingress-canary-kcm5w 1/1 Running 0 79m
openshift-ingress-canary ingress-canary-nsscv 1/1 Running 0 60m
openshift-ingress-operator ingress-operator-7c45957768-4cdbz 2/2 Running 4 (83m ago) 106m
openshift-ingress router-default-66ddcbc6d7-kmlxm 1/1 Running 4 (81m ago) 93m
openshift-ingress router-default-66ddcbc6d7-w2xh7 1/1 Running 4 (81m ago) 93m
openshift-insights insights-operator-7979dc4c88-cftjb 1/1 Running 1 (93m ago) 106m
openshift-kni-infra coredns-osc1-m0 2/2 Running 0 62m
openshift-kni-infra coredns-osc1-m1 2/2 Running 0 97m
openshift-kni-infra coredns-osc1-m2 2/2 Running 0 95m
openshift-kni-infra haproxy-osc1-m0 2/2 Running 0 62m
openshift-kni-infra haproxy-osc1-m1 2/2 Running 0 95m
openshift-kni-infra haproxy-osc1-m2 2/2 Running 0 94m
openshift-kni-infra keepalived-osc1-m0 2/2 Running 0 61m
openshift-kni-infra keepalived-osc1-m1 2/2 Running 0 96m
openshift-kni-infra keepalived-osc1-m2 2/2 Running 0 95m
openshift-kube-apiserver-operator kube-apiserver-operator-6fc8fddb4-lk26l 1/1 Running 2 (74m ago) 106m
openshift-kube-apiserver installer-3-osc1-m1 0/1 Error 0 91m
openshift-kube-apiserver installer-3-retry-1-osc1-m1 0/1 Completed 0 87m
openshift-kube-apiserver installer-6-osc1-m1 0/1 Completed 0 82m
openshift-kube-apiserver installer-8-osc1-m0 0/1 Completed 0 62m
openshift-kube-apiserver installer-8-osc1-m1 0/1 Completed 0 66m
openshift-kube-apiserver installer-8-osc1-m2 0/1 Error 0 78m
openshift-kube-apiserver installer-8-retry-1-osc1-m2 0/1 Completed 0 73m
openshift-kube-apiserver installer-9-osc1-m0 0/1 Completed 0 57m
openshift-kube-apiserver installer-9-osc1-m1 0/1 Completed 0 53m
openshift-kube-apiserver installer-9-osc1-m2 0/1 Completed 0 50m
openshift-kube-apiserver kube-apiserver-guard-osc1-m0 1/1 Running 0 59m
openshift-kube-apiserver kube-apiserver-guard-osc1-m1 1/1 Running 0 84m
openshift-kube-apiserver kube-apiserver-guard-osc1-m2 1/1 Running 0 71m
openshift-kube-apiserver kube-apiserver-osc1-m0 5/5 Running 0 54m
openshift-kube-apiserver kube-apiserver-osc1-m1 5/5 Running 0 51m
openshift-kube-apiserver kube-apiserver-osc1-m2 5/5 Running 0 48m
openshift-kube-controller-manager-operator kube-controller-manager-operator-76fccfd69d-7bg9k 1/1 Running 2 (74m ago) 107m
openshift-kube-controller-manager installer-5-osc1-m1 0/1 Completed 0 91m
openshift-kube-controller-manager installer-6-osc1-m2 0/1 Completed 0 84m
openshift-kube-controller-manager installer-8-osc1-m1 0/1 Error 0 81m
openshift-kube-controller-manager installer-8-retry-1-osc1-m1 0/1 Completed 0 79m
openshift-kube-controller-manager installer-9-osc1-m0 0/1 Completed 0 62m
openshift-kube-controller-manager installer-9-osc1-m1 0/1 Completed 0 71m
openshift-kube-controller-manager installer-9-osc1-m2 0/1 Completed 0 73m
openshift-kube-controller-manager kube-controller-manager-guard-osc1-m0 1/1 Running 0 59m
openshift-kube-controller-manager kube-controller-manager-guard-osc1-m1 1/1 Running 0 89m
openshift-kube-controller-manager kube-controller-manager-guard-osc1-m2 1/1 Running 0 82m
openshift-kube-controller-manager kube-controller-manager-osc1-m0 4/4 Running 0 60m
openshift-kube-controller-manager kube-controller-manager-osc1-m1 4/4 Running 3 (58m ago) 69m
openshift-kube-controller-manager kube-controller-manager-osc1-m2 4/4 Running 4 (60m ago) 71m
openshift-kube-controller-manager revision-pruner-8-osc1-m1 0/1 Completed 0 77m
openshift-kube-controller-manager revision-pruner-8-osc1-m2 0/1 Completed 0 77m
openshift-kube-controller-manager revision-pruner-9-osc1-m0 0/1 Completed 0 62m
openshift-kube-controller-manager revision-pruner-9-osc1-m1 0/1 Completed 0 73m
openshift-kube-controller-manager revision-pruner-9-osc1-m2 0/1 Completed 0 73m
openshift-kube-scheduler-operator openshift-kube-scheduler-operator-6dc4978c6-9mgt6 1/1 Running 2 (74m ago) 107m
openshift-kube-scheduler installer-5-osc1-m1 0/1 Completed 0 92m
openshift-kube-scheduler installer-6-osc1-m2 0/1 Completed 0 84m
openshift-kube-scheduler installer-7-osc1-m2 0/1 Completed 0 82m
openshift-kube-scheduler installer-8-osc1-m0 0/1 Completed 0 62m
openshift-kube-scheduler installer-8-osc1-m1 0/1 Completed 0 74m
openshift-kube-scheduler installer-8-osc1-m2 0/1 Error 0 81m
openshift-kube-scheduler installer-8-retry-1-osc1-m2 0/1 Completed 0 79m
openshift-kube-scheduler openshift-kube-scheduler-guard-osc1-m0 1/1 Running 0 60m
openshift-kube-scheduler openshift-kube-scheduler-guard-osc1-m1 1/1 Running 0 90m
openshift-kube-scheduler openshift-kube-scheduler-guard-osc1-m2 1/1 Running 0 83m
openshift-kube-scheduler openshift-kube-scheduler-osc1-m0 3/3 Running 0 60m
openshift-kube-scheduler openshift-kube-scheduler-osc1-m1 3/3 Running 0 70m
openshift-kube-scheduler openshift-kube-scheduler-osc1-m2 3/3 Running 0 78m
openshift-kube-scheduler revision-pruner-8-osc1-m0 0/1 Completed 0 62m
openshift-kube-scheduler revision-pruner-8-osc1-m1 0/1 Completed 0 69m
openshift-kube-scheduler revision-pruner-8-osc1-m2 0/1 Completed 0 69m
openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-54cf95684-h26pq 1/1 Running 3 (74m ago) 107m
openshift-kube-storage-version-migrator migrator-7d8bb67546-2sm6w 1/1 Running 0 95m
openshift-machine-api cluster-autoscaler-operator-864d6cfc55-sm4l9 2/2 Running 1 (74m ago) 106m
openshift-machine-api cluster-baremetal-operator-db79d6f9d-p9cq7 2/2 Running 0 106m
openshift-machine-api machine-api-controllers-84df7c787-w8lvs 7/7 Running 4 (74m ago) 92m
openshift-machine-api machine-api-operator-5964bd84bb-lgz2n 2/2 Running 2 (74m ago) 106m
openshift-machine-api metal3-7c4b6ff644-h5g66 7/7 Running 0 70m
openshift-machine-api metal3-image-cache-7xssd 1/1 Running 0 59m
openshift-machine-api metal3-image-cache-dv2zl 1/1 Running 0 90m
openshift-machine-api metal3-image-cache-qbr8r 1/1 Running 0 90m
openshift-machine-api metal3-image-customization-84d9b64d74-d45f9 1/1 Running 0 84m
openshift-machine-config-operator machine-config-controller-66dbbdc47b-rblj7 1/1 Running 2 (74m ago) 92m
openshift-machine-config-operator machine-config-daemon-4855k 2/2 Running 0 96m
openshift-machine-config-operator machine-config-daemon-cjtz4 2/2 Running 0 62m
openshift-machine-config-operator machine-config-daemon-xtzc8 2/2 Running 0 96m
openshift-machine-config-operator machine-config-operator-86948c978d-x5fmv 1/1 Running 2 (74m ago) 106m
openshift-machine-config-operator machine-config-server-bjfxs 1/1 Running 0 91m
openshift-machine-config-operator machine-config-server-cmx8l 1/1 Running 0 59m
openshift-machine-config-operator machine-config-server-grckw 1/1 Running 0 91m
openshift-marketplace certified-operators-9n582 1/1 Running 0 3m49s
openshift-marketplace community-operators-c88jm 1/1 Running 0 92m
openshift-marketplace marketplace-operator-d595b6d88-vv6gn 1/1 Running 3 (74m ago) 106m
openshift-marketplace redhat-marketplace-n2g9q 1/1 Running 0 92m
openshift-marketplace redhat-operators-8c5n4 1/1 Running 0 92m
openshift-monitoring alertmanager-main-0 6/6 Running 0 68m
openshift-monitoring alertmanager-main-1 6/6 Running 0 70m
openshift-monitoring cluster-monitoring-operator-7565f87c68-g9hr5 2/2 Running 0 106m
openshift-monitoring grafana-64bcbd55cb-vth9n 3/3 Running 0 78m
openshift-monitoring kube-state-metrics-5ff6bdc8c8-bfsx5 3/3 Running 0 91m
openshift-monitoring node-exporter-5d8xg 2/2 Running 0 91m
openshift-monitoring node-exporter-7g6x4 2/2 Running 0 62m
openshift-monitoring node-exporter-kfsvt 2/2 Running 0 91m
openshift-monitoring openshift-state-metrics-6cd845796d-nm2lk 3/3 Running 0 91m
openshift-monitoring prometheus-adapter-b6b78dc6f-nlt7k 1/1 Running 0 82m
openshift-monitoring prometheus-adapter-b6b78dc6f-tlzd7 1/1 Running 0 82m
openshift-monitoring prometheus-k8s-0 6/6 Running 0 67m
openshift-monitoring prometheus-k8s-1 6/6 Running 0 67m
openshift-monitoring prometheus-operator-68cdfd9bdf-49cpv 2/2 Running 0 92m
openshift-monitoring telemeter-client-cffc59c75-4s55j 3/3 Running 0 91m
openshift-monitoring thanos-querier-8b6847d46-2zqtw 6/6 Running 0 70m
openshift-monitoring thanos-querier-8b6847d46-gpzr5 6/6 Running 0 70m
openshift-multus multus-5mngz 1/1 Running 0 62m
openshift-multus multus-additional-cni-plugins-lxmzw 1/1 Running 0 97m
openshift-multus multus-additional-cni-plugins-qb268 1/1 Running 0 96m
openshift-multus multus-additional-cni-plugins-wb6dz 1/1 Running 0 62m
openshift-multus multus-admission-controller-5fnk2 2/2 Running 0 66m
openshift-multus multus-admission-controller-pcj9b 2/2 Running 0 60m
openshift-multus multus-admission-controller-qlh7h 2/2 Running 0 68m
openshift-multus multus-kmprm 1/1 Running 0 96m
openshift-multus multus-s8z5r 1/1 Running 0 97m
openshift-multus network-metrics-daemon-5kzv9 2/2 Running 0 97m
openshift-multus network-metrics-daemon-hdzlh 2/2 Running 0 96m
openshift-multus network-metrics-daemon-sfhkv 2/2 Running 0 62m
openshift-network-diagnostics network-check-source-c7f7bb99d-74v69 1/1 Running 0 96m
openshift-network-diagnostics network-check-target-n8hwq 1/1 Running 0 62m
openshift-network-diagnostics network-check-target-pz5tm 1/1 Running 0 96m
openshift-network-diagnostics network-check-target-qr5zv 1/1 Running 0 96m
openshift-network-operator network-operator-5754c48fbc-k44ds 1/1 Running 3 (74m ago) 107m
openshift-oauth-apiserver apiserver-5876b7bcb5-2vwbs 1/1 Running 0 56m
openshift-oauth-apiserver apiserver-5876b7bcb5-7hp84 1/1 Running 0 54m
openshift-oauth-apiserver apiserver-5876b7bcb5-xbjj8 1/1 Running 0 55m
openshift-operator-lifecycle-manager catalog-operator-5bf85b9b85-zhwvw 1/1 Running 0 106m
openshift-operator-lifecycle-manager collect-profiles-28121265-4hbwx 0/1 Completed 0 32m
openshift-operator-lifecycle-manager collect-profiles-28121280-784wz 0/1 Completed 0 17m
openshift-operator-lifecycle-manager collect-profiles-28121295-qnzrg 0/1 Completed 0 2m7s
openshift-operator-lifecycle-manager olm-operator-6d9469f8c6-58btp 1/1 Running 0 106m
openshift-operator-lifecycle-manager package-server-manager-7cfd46756f-mpbpf 1/1 Running 3 (74m ago) 106m
openshift-operator-lifecycle-manager packageserver-56d9774779-7b5lc 1/1 Running 0 93m
openshift-operator-lifecycle-manager packageserver-56d9774779-vcfwn 1/1 Running 0 93m
openshift-sdn sdn-8xczz 2/2 Running 0 62m
openshift-sdn sdn-controller-7zvwp 2/2 Running 2 (74m ago) 96m
openshift-sdn sdn-controller-g4qt5 2/2 Running 0 96m
openshift-sdn sdn-controller-kbbnm 2/2 Running 0 62m
openshift-sdn sdn-fj5fg 2/2 Running 0 96m
openshift-sdn sdn-xgsth 2/2 Running 0 96m
openshift-service-ca-operator service-ca-operator-bf9df4594-6gd75 1/1 Running 2 (86m ago) 106m
openshift-service-ca service-ca-7774df9c5-6z92l 1/1 Running 2 (74m ago) 95m
Before installing F5 XC Kubernetes CE Pod based on https://docs.cloud.f5.com/docs/how-to/site-management/create-k8s-site, Hugepages and a storage class for dynamic PVC named 'standard (default)' are required.
To enable hugepages on worker nodes (in this 3 node setup, the master nodes are also labeled as worker), follow the official instructions at https://docs.openshift.com/container-platform/4.10/scalability_and_performance/what-huge-pages-do-and-how-they-are-consumed-by-apps.html
Couldn't find a way to enable hugepages on master nodes enabled as worker nodes. Accrding to https://docs.openshift.com/container-platform/4.10/scalability_and_performance/what-huge-pages-do-and-how-they-are-consumed-by-apps.html#at-boot-time this is only supported on worker nodes. So I ended up adding 3 worker nodes using the Openshift Clusters UI -> Select cluster -> Add Hosts tab and create a new ISO and use it to boot up 3 additional VMs.
In a nutshell, it requires to label the worker nodes with 'worker-hp' (for huegapges) and apply MachineConfigPool and Tuned:
- Label nodes
$ k get nodes NAME STATUS ROLES AGE VERSION
osc1-m0 Ready master,worker 70m v1.23.17+16bcd69
osc1-m1 Ready master,worker 105m v1.23.17+16bcd69
osc1-m2 Ready master,worker 103m v1.23.17+16bcd69
$ oc label node osc1-m0 node-role.kubernetes.io/worker-hp=
node/osc1-m0 labeled
$ oc label node osc1-m1 node-role.kubernetes.io/worker-hp=
node/osc1-m1 labeled
$ oc label node osc1-m2 node-role.kubernetes.io/worker-hp=
node/osc1-m2 labeled
$ k get nodes
NAME STATUS ROLES AGE VERSION
osc1-m0 Ready master,worker,worker-hp 70m v1.23.17+16bcd69
osc1-m1 Ready master,worker,worker-hp 105m v1.23.17+16bcd69
osc1-m2 Ready master,worker,worker-hp 104m v1.23.17+16bcd69
- Apply hugepages-tuned-boottime.yaml
$ cat hugepages-tuned-boottime.yaml
apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
name: hugepages
namespace: openshift-cluster-node-tuning-operator
spec:
profile:
- data: |
[main]
summary=Boot time configuration for hugepages
include=openshift-node
[bootloader]
cmdline_openshift_node_hugepages=hugepagesz=2M hugepages=400
name: openshift-node-hugepages
recommend:
- machineConfigLabels:
machineconfiguration.openshift.io/role: "worker-hp"
priority: 30
profile: openshift-node-hugepages
$ oc apply -f hugepages-tuned-boottime.yaml
tuned.tuned.openshift.io/hugepages created
- Apply hugepages-mcp.yaml
$ cat hugepages-mcp.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
name: worker-hp
labels:
worker-hp: ""
spec:
machineConfigSelector:
matchExpressions:
- {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-hp]}
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker-hp: ""
$ oc apply -f hugepages-mcp.yaml
machineconfigpool.machineconfiguration.openshift.io/worker-hp created
Add worker nodes (couldn't add hugepages to master nodes) and label them:
$ oc label node osc1-w0 node-role.kubernetes.io/worker-hp=
node/osc1-w0 labeled
$ oc label node osc1-w1 node-role.kubernetes.io/worker-hp=
node/osc1-w1 labeled
$ oc label node osc1-w2 node-role.kubernetes.io/worker-hp=
node/osc1-w2 labeled
Check hugepages after a few minutes:
$ cat check-hp.sh
#!/bin/bash
nodes=$(kubectl get nodes -o json| jq -r '.items[].metadata.name')
echo $nodes ...
for node in $nodes; do
echo -n "$node "
oc get node $node -o jsonpath="{.status.allocatable.hugepages-2Mi}"
echo ""
done
$ ./check-hp.sh
$ ./check-hp.sh
osc1-m0 osc1-m1 osc1-m2 osc1-w0 osc1-w1 osc1-w2 ...
osc1-m0 0
osc1-m1 0
osc1-m2 0
osc1-w0 800Mi
osc1-w1 800Mi
osc1-w2 800Mi
The F5 XC Kubernetes CE pod requires Hugepages
For this setup, the following NFS server is used:
$ showmount -e 192.168.42.1
Exports list on 192.168.42.1:
/opt/nfs_shared_folder 192.168.0.0/16
Note on testing NFS mount on OSX:
Requires -o resvport according to, which worked in my case:
https://thornelabs.net/posts/operation-not-permitted-mounting-nfs-share-on-os-x-mountain-lion/
$sudo mount -t nfs -o resvport 192.168.42.1:/opt/nfs_shared_folder /private/nfs/
Check first if there is an existing storage class, ready for use (not the case with assisted installer, but true for ROSA, Redhat OpenShift on AWS):
$ oc get sc
No resources found
Use files from https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner/deploy/
- create rbac
$ oc apply -f rbac.yaml
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
- Deploy the NFS-client provisioner
Modify nfs-subdir-external-provisioner/deploy/deployment.yaml with the IP and path to an existing NFS server, then apply:
$ oc apply -f deployment.yaml
deployment.apps/nfs-client-provisioner created
- Create storage-class
standard (default)
Modify the name and set is-default-class to true in nfs-subdir-external-provisioner/deploy/class.yaml, then apply:
$ cat class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: standard
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
archiveOnDelete: "false"
Create storage-class:
$ oc apply -f class.yaml
storageclass.storage.k8s.io/standard created
Check:
$ oc get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
standard (default) k8s-sigs.io/nfs-subdir-external-provisioner Delete Immediate false 2m5s
Important: The name of the storage-class must be exactly standard (default) in order for kubernetes CE pod to deploy.
Follow instructions on https://docs.cloud.f5.com/docs/how-to/site-management/create-k8s-site
Download and adjust https://gitlab.com/volterra.io/volterra-ce/-/blob/master/k8s/ce_k8s.yml, then apply it
$ oc apply -f ce_k8s_osc1.yaml
namespace/ves-system created
serviceaccount/volterra-sa created
role.rbac.authorization.k8s.io/volterra-admin-role created
rolebinding.rbac.authorization.k8s.io/volterra-admin-role-binding created
daemonset.apps/volterra-ce-init created
serviceaccount/vpm-sa created
role.rbac.authorization.k8s.io/vpm-role created
clusterrole.rbac.authorization.k8s.io/vpm-cluster-role created
rolebinding.rbac.authorization.k8s.io/vpm-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/vpm-sa created
clusterrolebinding.rbac.authorization.k8s.io/ver created
configmap/vpm-cfg created
statefulset.apps/vp-manager created
service/vpm created
service/ver-nodeport-ver-0 created
service/ver-nodeport-ver-1 created
service/ver-nodeport-ver-2 created
Check progress via kubectl/oc and accept pending registration via F5 XC UI.
$ oc get pods -n ves-system
NAME READY STATUS RESTARTS AGE
volterra-ce-init-2d246 1/1 Running 0 2m55s
volterra-ce-init-bfqmh 1/1 Running 0 2m55s
volterra-ce-init-klvwc 1/1 Running 0 2m55s
volterra-ce-init-nts6k 1/1 Running 0 2m55s
volterra-ce-init-nvtl9 1/1 Running 0 2m55s
volterra-ce-init-s8nd7 1/1 Running 0 2m55s
vp-manager-0 1/1 Running 0 28s
vp-manager-1 1/1 Running 0 58s
vp-manager-2 1/1 Running 0 118s
Accept the pending registrations in the F5 XC UI, changing cluster size to 3.
This will trigger the installation of ver-# pods in the same ves-system namespace. After some time, the CE pod is successfully deployed:
$ oc get pods -o wide -n ves-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
etcd-0 2/2 Running 0 27m 10.128.2.9 osc1-w1 <none> <none>
etcd-1 2/2 Running 0 27m 10.129.2.10 osc1-w2 <none> <none>
etcd-2 2/2 Running 0 27m 10.131.0.78 osc1-w0 <none> <none>
prometheus-845ddc5ccc-gxcgx 5/5 Running 0 26m 10.129.2.11 osc1-w2 <none> <none>
ver-0 17/17 Running 0 26m 10.128.2.10 osc1-w1 <none> <none>
ver-1 17/17 Running 0 19m 10.131.0.80 osc1-w0 <none> <none>
ver-2 17/17 Running 0 12m 10.129.2.12 osc1-w2 <none> <none>
volterra-ce-init-2d246 1/1 Running 0 31m 192.168.40.35 osc1-w0 <none> <none>
volterra-ce-init-bfqmh 1/1 Running 0 31m 192.168.40.47 osc1-w1 <none> <none>
volterra-ce-init-klvwc 1/1 Running 0 31m 192.168.40.70 osc1-m0 <none> <none>
volterra-ce-init-nts6k 1/1 Running 0 31m 192.168.40.46 osc1-w2 <none> <none>
volterra-ce-init-nvtl9 1/1 Running 0 31m 192.168.40.31 osc1-m2 <none> <none>
volterra-ce-init-s8nd7 1/1 Running 0 31m 192.168.40.62 osc1-m1 <none> <none>
vp-manager-0 1/1 Running 2 (27m ago) 29m 10.129.2.9 osc1-w2 <none> <none>
vp-manager-1 1/1 Running 2 (26m ago) 29m 10.128.2.8 osc1-w1 <none> <none>
vp-manager-2 1/1 Running 3 (4m58s ago) 30m 10.131.0.77 osc1-w0 <none> <none>
$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
osc1-m0 Ready master,worker,worker-hp 20h v1.23.17+16bcd69 192.168.40.70 <none> Red Hat Enterprise Linux CoreOS 410.84.202305261556-0 (Ootpa) 4.18.0-305.88.1.el8_4.x86_64 cri-o://1.23.5-15.rhaos4.10.git0bbb0d9.el8
osc1-m1 Ready master,worker,worker-hp 21h v1.23.17+16bcd69 192.168.40.62 <none> Red Hat Enterprise Linux CoreOS 410.84.202305261556-0 (Ootpa) 4.18.0-305.88.1.el8_4.x86_64 cri-o://1.23.5-15.rhaos4.10.git0bbb0d9.el8
osc1-m2 Ready master,worker,worker-hp 21h v1.23.17+16bcd69 192.168.40.31 <none> Red Hat Enterprise Linux CoreOS 410.84.202305261556-0 (Ootpa) 4.18.0-305.88.1.el8_4.x86_64 cri-o://1.23.5-15.rhaos4.10.git0bbb0d9.el8
osc1-w0 Ready worker,worker-hp 17h v1.23.17+16bcd69 192.168.40.35 <none> Red Hat Enterprise Linux CoreOS 410.84.202305261556-0 (Ootpa) 4.18.0-305.88.1.el8_4.x86_64 cri-o://1.23.5-15.rhaos4.10.git0bbb0d9.el8
osc1-w1 Ready worker,worker-hp 17h v1.23.17+16bcd69 192.168.40.47 <none> Red Hat Enterprise Linux CoreOS 410.84.202305261556-0 (Ootpa) 4.18.0-305.88.1.el8_4.x86_64 cri-o://1.23.5-15.rhaos4.10.git0bbb0d9.el8
osc1-w2 Ready worker,worker-hp 17h v1.23.17+16bcd69 192.168.40.46 <none> Red Hat Enterprise Linux CoreOS 410.84.202305261556-0 (Ootpa) 4.18.0-305.88.1.el8_4.x86_64 cri-o://1.23.5-15.rhaos4.10.git0bbb0d9.el8