PoC for Single Node OpenShift plus Worker

README.md

PoC for Single Node OpenShift plus Worker

Single Node OpenShift (SNO) as of 4.9 is a Dev Preview and adding an additional worker is out of scope for now.

The notes listed here are inspirational and totally unsupported. The PoC was heavily inspiried by https://medium.com/codex/openshift-single-node-for-distributed-clouds-582f84022bd0

If interested, please follow, vote and/or contribute to OCPPLAN-6839 Single replica control plane topology expansion.

Motivation

Save resources on the control plane side. The control node shall run all the infra components as well. Have the worker completely dedicated to SAP HANA on CNV validation.

Scenario and prerequisites

• 2 bare metal hosts
• jumpbox
• external DNS server
• static IP configuration of hosts
• http server (in this example the same as jumpbox)
  • serves ignition files and ISOs
• no loadbalancer
• no PXE nor DHCP server
• no AI (Assisted Installer) utilized (not recommended)

Process

DNS configuration

See the required DNS records: https://docs.openshift.com/container-platform/4.9/installing/installing_sno/install-sno-preparing-to-install-sno.html#requirements-for-installing-on-a-single-node_install-sno-preparing

In our case, all of them point to the control host (SNO). The following is the snippet from dnsmasq configuration:

host-record=ctrl.snoplusone.ocp.vslen,10.123.223.23
host-record=worker.snoplusone.ocp.vslen,10.123.223.36

# just for the convenience
cname=ctrl,ctrl.snoplusone.ocp.vslen
cname=worker,worker.snoplusone.ocp.vslen

host-record=snoplusone.ocp.vslen,10.123.223.23
host-record=api.snoplusone.ocp.vslen,10.123.223.23
host-record=api-int.snoplusone.ocp.vslen,10.123.223.23
address=/apps.snoplusone.ocp.vslen/10.123.223.23

Note the IP address of the DNS server (in our case 10.1.2.232).

Prepare static IP network configuration file for SNO

Take ./eno1.nmconnection as an example. Prepare the same iface fail both for control and worker node.

For an unknown reason, the network configuration did not work for worker node, thus only one connection files was needed.

Note you interface name may differ. You may want to initially boot to the live ISO just to see the interface name of the interface you want to configure.

Generate ISOs

1. Download RHCOS 4.9 live ISO from https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.9/latest/

2. Download openshift-install, oc clients and coreos-installer as described at https://medium.com/codex/openshift-single-node-for-distributed-clouds-582f84022bd0#8ab9

   • in addition to that, make sure that "jq >= 1.6" and "moreutils" are installed and available on your PATH in jumpbox

3. Download ./install-config.yaml and update it for the pullSecret and sshKey. Set the desired domain and cluster names. See more at Required configuration parameters. Get your own pullSecret from Red Hat Hybrid Cloud Console.

4. Download ./deploy.sh script and edit the variables at the beginning (ISO location, connection file names, etc.).

5. Run it.

Provision SNO

1. Logon to the BMC of control host, mount the 4.9-ctrl.iso (from your http location) and boot the CD/DVD virtual media.

2. Wait for installation to complete. See Booting the SNO from ISO for more information.

3. Make sure the ingress controller stays on tho control node before a worker is added:

    oc patch ingresscontroller/default -n openshift-ingress-operator --type merge -p '{"spec":{
            "nodePlacement": { 
                "nodeSelector": {
                    "matchLabels": {
                        "node-role.kubernetes.io/master": ""
        }}}}}'
   

   Otherwise, you will need to setup a load balancer.

Add a worker

Once the SNO is up and running, you can start provisioning the worker.

The worker.ign has been generated earlier by the deploy.sh script. Booting the ISO with the embedded ignition should work, but it did not in our case. Therefore, in this example, we will boot into the live ISO image and kick off the installation from there manually.

1. Logon to the BMC of the worker host, mount the 4.9-worker.iso (from your http location) and boot the CD/DVD virtual media.

2. Wait until you end up in shell.

3. Configure the network interface, for example:

    id='Wired connection 1'
    iface=eno24
    newId="$iface"
    sudo nmcli c down "$id"
    sudo nmcli c modify "$id" ipv4.addresses "10.123.223.36/22" \
        ipv4.gateway 10.97.228.1 ipv4.DNS 10.1.2.232 \
        ipv4.method manual connection.id "$newId"
    sudo nmcli c up "$newId"
   

4. Kick off the installation:

    sudo coreos-installer install --copy-network --ignition-url \
        http://10.1.2.232:8080/rhcos/snoplusone/ignition/worker.ign \
        --insecure --insecure-ignition /dev/sda
   

5. Once done, reboot and unmount the ISO.

6. The worker will generate CSR for joining the SNO cluster that needs to be approved manually. You can wait for it and approve automatically like this:

    while true; do
        oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | \
            xargs -r oc adm certificate approve
        sleep 10;
    done
   

7. Profit!

deploy.sh

#!/usr/bin/env bash

set -euo pipefail
IFS=$'\n\t'

SNO_NM_CONNECTION_FILE=./eno1.nmconnection
#WORKER_NM_CONNECTION_FILE=./sap-eno24.nmconnection
ISO=./rhcos-4.9.0-x86_64-live.x86_64.iso
# API will be available at https://api.$CLUSTER_NAME.$DOMAIN:6443
CLUSTER_NAME=snoplusone
DOMAIN=ocp.vslen
# assuming http server runs on the same host this script is run
HTTP_SERVE_ROOT=/srv/html/rhcos
IMAGE_SERVE="$HTTP_SERVE_ROOT/images"
IGNITION_SERVE="$HTTP_SERVE_ROOT/$CLUSTER_NAME/ignition"

function patchIgnitionFile() {
    local nmconfile="$1"
    local ignFn="$2"
    local content
    content="$(base64 -w0 <"$nmconfile")"
    jq --arg ifcfgContent "$content" --arg ifcfgFileName "$(basename "$nmconfile")" \
        '.storage |= (.files |= ((. // []) + [{
                "contents": {
                  "source": ("data:text/plain;charset=utf-8;base64," + $ifcfgContent),
                  "verification": {}
                },
                "filesystem": "root",
                "mode": 384,
                "path": ("/etc/NetworkManager/system-connections/" + $ifcfgFileName)
            }]))' "$ignFn" | sponge "$ignFn"
}

function getRootCA() {
    local bootstrapIgn="$1"
    jq -r '.storage.files[] | select(.path == "/opt/openshift/tls/root-ca.crt") | .contents.source' \
        "$bootstrapIgn" | sed 's#^data:text/plain;charset=utf-8;base64,##' | base64 -d
}

function genWorkerIgn() {
    local bootstrapIgn="$1"
    local rootCA
    rootCA="$(getRootCA "$bootstrapIgn")"
    jq -n --arg rootCA "$rootCA" --arg clusterName "$CLUSTER_NAME" --arg domain "$DOMAIN" '{
          "ignition": {
            "config": {
              "merge": [
                {
                  "source": ("https://api-int." + $clusterName +"."+ $domain + ":22623/config/worker")
                }
              ]
            },
            "security": {
              "tls": {
                "certificateAuthorities": [
                  {
                    "source": ("data:text/plain;charset=utf-8;base64," + ($rootCA | @base64))
                  }
                ]
              }
            },
            "version": "3.1.0"
          }
        }'
}


if [[ -e "$CLUSTER_NAME" ]]; then
    rm -rf ./"$CLUSTER_NAME"
    echo -n ''
fi
mkdir "$CLUSTER_NAME"
ln -vf install-config.yaml "$CLUSTER_NAME"/
cp -a install-config.yaml "$CLUSTER_NAME"/install-config.bak.yaml
openshift-install create manifests --dir=./"$CLUSTER_NAME"
openshift-install create single-node-ignition-config --dir=./"$CLUSTER_NAME"
patchIgnitionFile "$SNO_NM_CONNECTION_FILE" ./"$CLUSTER_NAME"/bootstrap-in-place-for-live-iso.ign
cp -v ./"$CLUSTER_NAME"/bootstrap-in-place-for-live-iso.ign "$IGNITION_SERVE"/
cp -v "$ISO" "$CLUSTER_NAME"-4.9-ctrl.iso
coreos-installer iso ignition embed -fi ./"$CLUSTER_NAME"/bootstrap-in-place-for-live-iso.ign \
    "$CLUSTER_NAME"-4.9-ctrl.iso
mv -v "$CLUSTER_NAME"-4.9-ctrl.iso "$IMAGE_SERVE/"

# generate worker files
genWorkerIgn ./"$CLUSTER_NAME"/bootstrap-in-place-for-live-iso.ign >./"$CLUSTER_NAME"/worker.ign
# For an unknown reason, the network configuration file did not work in our case.
    #patchIgnitionFile "$WORKER_NM_CONNECTION_FILE" ./"$CLUSTER_NAME"/worker.ign
    #cp -v "$ISO" "$CLUSTER_NAME"-4.9-worker.iso
    #coreos-installer iso ignition embed -fi ./"$CLUSTER_NAME"/worker.ign \
    #    "$CLUSTER_NAME"-4.9-worker.iso
    #mv -v "$CLUSTER_NAME"-4.9-worker.iso "$IGNITION_SERVE"/
cp -v "$ISO" "$IMAGE_SERVE/$CLUSTER_NAME"-4.9-worker.iso
cp -v ./"$CLUSTER_NAME"/worker.ign "$IGNITION_SERVE"/

eno1.nmconnection

[connection]
id=eno1
uuid=972e9e95-840d-38a3-a904-58fadda8c5f2
type=ethernet
autoconnect-priority=-999
interface-name=eno1
permissions=
timestamp=1636451317

[ethernet]
mac-address-blacklist=

[ipv4]
address1=10.123.223.23/22,10.97.228.1
dns=10.1.2.232;
dns-search=snoplusone.ocp.vslen,ocp.vslen
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dns-search=snoplusone.ocp.vslen,ocp.vslen
method=auto

[proxy]

install-config.yaml

apiVersion: v1
baseDomain: ocp.vslen
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 1
metadata:
  name: snoplusone
bootstrapInPlace:
  # awful hack to copy static ip configuration to the final disk image
  # the arguments are passed verbatim to the `coreos-installer install`
  installationDisk: "--copy-network /dev/sda"
platform:
  none: {}
pullSecret: "CHANGEME"
sshKey: "CHANGEME"