Install Rancher 2.0 HA by using self signed certificates (+ intermediate) and Layer 4 Loadbalancer (TCP)

rancher2ha_selfsigned_layer4lb.md

Install Rancher 2.0 HA by using self signed certificates (+ intermediate) and Layer 4 Loadbalancer (TCP)

Warning: the RKE install method is only supported up to v2.0.8!

This gist describes how to setup Rancher 2 HA, by using self signed certificates (with intermediate) and a Layer 4 Loadbalancer (TCP)

Requirements

• Linux OS
• OS Binaries
  • wget
  • openssl
  • base64
  • sed
• RKE (https://github.com/rancher/rke/releases/latest)
• kubectl (https://kubernetes.io/docs/tasks/tools/install-kubectl/)

Variables

Variables used in this guide:

• FQDN: rancher.yourdomain.com
• Node 1 IP: 192.168.0.1
• Node 2 IP: 192.168.0.2
• Node 3 IP: 192.168.0.3

Create self signed certificates

This script is based on https://jamielinux.com/docs/openssl-certificate-authority/index.html, which creates a Root CA, Intermediate CA and Certificate.

Run this script and follow the instructions, you need to fill in:

• Common Name for the Root CA (Yourcompany Root CA for example)
• Common Name for the Intermediate (Yourcompany Intermediate CA for example)
• Use your FQDN as Common Name for the certificate (rancher.yourdomain.com)

Copy the contents below or download via https://gist.githubusercontent.com/superseb/ce1f544eed8e793ad862c9bcfe3d19fe/raw/fca5a0c3c74ed5f7f9755b7777f31ae4faff0bd1/cert.sh

#!/bin/bash

# Root pair
mkdir /root/ca
cd /root/ca
mkdir certs crl newcerts private
chmod 700 private
touch index.txt
echo 1000 > serial

wget -O /root/ca/openssl.cnf https://jamielinux.com/docs/openssl-certificate-authority/_downloads/root-config.txt

echo "##########"
echo "CREATE root key"
echo "##########"
openssl genrsa -aes256 -out private/ca.key.pem 4096
chmod 400 private/ca.key.pem

echo "##########"
echo "CREATE root certificate"
echo "Fill in the Common Name!"
echo "##########"
openssl req -config openssl.cnf \
      -key private/ca.key.pem \
      -new -x509 -days 7300 -sha256 -extensions v3_ca \
      -out certs/ca.cert.pem
      
chmod 444 certs/ca.cert.pem

# Intermediate
mkdir /root/ca/intermediate
cd /root/ca/intermediate
mkdir certs crl csr newcerts private
chmod 700 private
touch index.txt
echo 1000 > serial
echo 1000 > /root/ca/intermediate/crlnumber

wget -O /root/ca/intermediate/openssl.cnf https://jamielinux.com/docs/openssl-certificate-authority/_downloads/intermediate-config.txt
echo "##########"
echo "KEY intermediate"
echo "##########"
cd /root/ca
openssl genrsa -aes256 \
      -out intermediate/private/intermediate.key.pem 4096
chmod 400 intermediate/private/intermediate.key.pem

echo "##########"
echo "CSR intermediate"
echo "Fill in the Common Name!"
echo "##########"
openssl req -config intermediate/openssl.cnf -new -sha256 \
      -key intermediate/private/intermediate.key.pem \
      -out intermediate/csr/intermediate.csr.pem

echo "##########"
echo "SIGN intermediate"
echo "##########"
openssl ca -config openssl.cnf -extensions v3_intermediate_ca \
      -days 3650 -notext -md sha256 \
      -in intermediate/csr/intermediate.csr.pem \
      -out intermediate/certs/intermediate.cert.pem
      
chmod 444 intermediate/certs/intermediate.cert.pem

cat intermediate/certs/intermediate.cert.pem \
      certs/ca.cert.pem > intermediate/certs/ca-chain.cert.pem
chmod 444 intermediate/certs/ca-chain.cert.pem

echo "##########"
echo "KEY certificate"
echo "##########"
openssl genrsa -aes256 \
      -out intermediate/private/rancher.yourdomain.com.key.pem 2048
chmod 400 intermediate/private/rancher.yourdomain.com.key.pem

echo "##########"
echo "CSR certificate"
echo "Use rancher.yourdomain.com as Common Name"
echo "##########"
openssl req -config intermediate/openssl.cnf \
      -key intermediate/private/rancher.yourdomain.com.key.pem \
      -new -sha256 -out intermediate/csr/rancher.yourdomain.com.csr.pem
      
echo "##########"
echo "SIGN certificate"
echo "##########"
openssl ca -config intermediate/openssl.cnf \
      -extensions server_cert -days 375 -notext -md sha256 \
      -in intermediate/csr/rancher.yourdomain.com.csr.pem \
      -out intermediate/certs/rancher.yourdomain.com.cert.pem
chmod 444 intermediate/certs/rancher.yourdomain.com.cert.pem

echo "##########"
echo "Create files to be used for Rancher"
echo "##########"
mkdir -p /root/ca/rancher/base64
cp /root/ca/certs/ca.cert.pem /root/ca/rancher/cacerts.pem
cat /root/ca/intermediate/certs/rancher.yourdomain.com.cert.pem /root/ca/intermediate/certs/intermediate.cert.pem > /root/ca/rancher/cert.pem
echo "##########"
echo "Removing passphrase from Rancher certificate key"
echo "##########"
openssl rsa -in /root/ca/intermediate/private/rancher.yourdomain.com.key.pem -out /root/ca/rancher/key.pem
cat /root/ca/rancher/cacerts.pem | base64 -w0 > /root/ca/rancher/base64/cacerts.base64
cat /root/ca/rancher/cert.pem | base64 -w0 > /root/ca/rancher/base64/cert.base64
cat /root/ca/rancher/key.pem | base64 -w0 > /root/ca/rancher/base64/key.base64

echo "##########"
echo "Verify certificates"
echo "##########"
openssl verify -CAfile certs/ca.cert.pem \
      intermediate/certs/intermediate.cert.pem
openssl verify -CAfile intermediate/certs/ca-chain.cert.pem \
      intermediate/certs/rancher.yourdomain.com.cert.pem

The script should end the following, indicating everything is OK.

##########
Verify certificates
##########
intermediate/certs/intermediate.cert.pem: OK
intermediate/certs/rancher.yourdomain.com.cert.pem: OK

Configure the RKE template

This guide is based on the 3-node-certificate.yml template, which is used for self signed certificates and using a Layer 4 Loadbalancer (TCP).

Download the template

wget -O /root/3-node-certificate.yml https://raw.githubusercontent.com/rancher/rancher/master/rke-templates/3-node-certificate.yml

Edit the values (FQDN, BASE64_CRT, BASE64_KEY, BASE64_CA)

This command will replace the values, you only have to edit rancher.yourdomain.com below to match your FQDN.

• Change rancher.yourdomain.com to your FQDN

sed -i -e "s/<FQDN>/rancher.yourdomain.com/" -e "s/<BASE64_CRT>/$(cat /root/ca/rancher/base64/cert.base64)/" -e "s/<BASE64_KEY>/$(cat /root/ca/rancher/base64/key.base64)/" -e "s/<BASE64_CA>/$(cat /root/ca/rancher/base64/cacerts.base64)/" /root/3-node-certificate.yml

Validate that the FQDN is replaced correctly:

cat 3-node-certificate.yml | grep rancher.yourdomain.com

Configure nodes

On top of the 3-node-certificate.yml file, configure your nodes that will be used for the cluster.

Example:

nodes:
  - address: 192.168.0.1 # hostname or IP to access nodes
    user: root
    role: [controlplane,etcd,worker] # K8s roles for node
    ssh_key_path: ~/.ssh/id_rsa # path to PEM file
  - address: 192.168.0.2
    user: root
    role: [controlplane,etcd,worker]
    ssh_key_path: ~/.ssh/id_rsa # path to PEM file
  - address: 192.168.0.3
    user: root
    role: [controlplane,etcd,worker]
    ssh_key_path: ~/.ssh/id_rsa # path to PEM file

Run RKE to setup the cluster

Run RKE to setup the cluster

./rke_linux-amd64 up --config 3-node-certificate.yml

Which should finish with the following to indicate that it is successfull:

INFO[0XXX] Finished building Kubernetes cluster successfully

Validate cluster

Nodes

All nodes should be in Ready status (it can take a few minutes before they get Ready)

kubectl --kubeconfig kube_config_3-node-certificate.yml get nodes
NAME              STATUS    ROLES                      AGE       VERSION
192.168.0.1       Ready     controlplane,etcd,worker   1m        v1.10.5
192.168.0.2       Ready     controlplane,etcd,worker   1m        v1.10.5
192.168.0.3       Ready     controlplane,etcd,worker   1m        v1.10.5

Pods

All pods must be in Running status, and names containing job should be in Completed status.

kubectl --kubeconfig kube_config_3-node-certificate.yml get pods --all-namespaces
NAMESPACE       NAME                                      READY     STATUS      RESTARTS   AGE
cattle-system   cattle-859b6cdc6b-qzs2z                   1/1       Running     0          3m
ingress-nginx   default-http-backend-564b9b6c5b-rqcvk     1/1       Running     0          4m
ingress-nginx   nginx-ingress-controller-4vmf7            1/1       Running     0          4m
ingress-nginx   nginx-ingress-controller-ftm6c            1/1       Running     0          4m
ingress-nginx   nginx-ingress-controller-vs44t            1/1       Running     0          4m
kube-system     canal-b9gvb                               3/3       Running     0          4m
kube-system     canal-h8wcb                               3/3       Running     0          4m
kube-system     canal-n6qtt                               3/3       Running     0          4m
kube-system     kube-dns-5ccb66df65-zjbrt                 3/3       Running     0          4m
kube-system     kube-dns-autoscaler-6c4b786f5-7q657       1/1       Running     0          4m
kube-system     rke-ingress-controller-deploy-job-d64ch   0/1       Completed   0          4m
kube-system     rke-kubedns-addon-deploy-job-hbz44        0/1       Completed   0          4m
kube-system     rke-network-plugin-deploy-job-g5wtq       0/1       Completed   0          4m
kube-system     rke-user-addon-deploy-job-9c5j5           0/1       Completed   0          4m

Ingress created

The created Ingress should match your FQDN

kubectl --kubeconfig kube_config_3-node-certificate.yml get ingress -n cattle-system

Overlay network

To test the overlay network:

Save this file as ds-alpine.yml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: alpine
spec:
  selector:
      matchLabels:
        name: alpine
  template:
    metadata:
      labels:
        name: alpine
    spec:
      tolerations:
      - effect: NoExecute
        key: "node-role.kubernetes.io/etcd"
        value: "true"
      - effect: NoSchedule
        key: "node-role.kubernetes.io/controlplane"
        value: "true"
      containers:
      - image: alpine
        imagePullPolicy: Always
        name: alpine
        command: ["sh", "-c", "tail -f /dev/null"]
        terminationMessagePath: /dev/termination-log

Run the following command:

kubectl --kubeconfig kube_config_3-node-certificate.yml create -f ds-alpine.yml
kubectl --kubeconfig kube_config_3-node-certificate.yml rollout status ds/alpine -w

Wait until it returns: daemon set "alpine" successfully rolled out.

Then execute the following command to test network connectivity:

echo "=> Start"; kubectl --kubeconfig kube_config_3-node-certificate.yml get pods -l name=alpine -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.nodeName}{"\n"}{end}' | while read spod shost; do kubectl --kubeconfig kube_config_3-node-certificate.yml get pods -l name=alpine -o jsonpath='{range .items[*]}{@.status.podIP}{" "}{@.spec.nodeName}{"\n"}{end}' | while read tip thost; do kubectl --kubeconfig kube_config_3-node-certificate.yml --request-timeout='10s' exec $spod -- /bin/sh -c "ping -c2 $tip > /dev/null 2>&1"; RC=$?; if [ $RC -ne 0 ]; then echo $shost cannot reach $thost; fi; done; done; echo "=> End"

Which should result in the following, indicating it is working as expected:

=> Start
=> End

Validate Rancher

Run the following to validate the accessibility to Rancher:

Validate certificates

To validate the certificates:

openssl s_client -CAfile /root/ca/rancher/cacerts.pem -connect 192.168.0.1:443 -servername rancher.yourdomain.com

This should result in the following, indicating the chain is correct. You can repeat this for the other hosts (192.168.0.2 and 192.168.0.3)

    Start Time: 1531422707
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)

Validate connection

Use the following command to see if you can reach the Rancher server:

curl --cacert /root/ca/rancher/cacerts.pem --resolve rancher.yourdomain.com:443:192.168.0.1 https://rancher.yourdomain.com/ping

Response should be pong.

Set up loadbalancer

If this is all functioning correctly, you can put a load balancer in front. See for an NGINX example: https://rancher.com/docs/rancher/v2.x/en/installation/ha-server-install/#2-configure-load-balancer

How to install rancher using helm chart along with the self-signed certificate. I have followed the rancher2 documentation but that is not helping me to resolve the certificates. I have successfully installed Rancher server on 3 nodes. When I tried to Add cluster to one of the cluster node with etcd and control plane, the cluster node itself complaing about the x509: certificate signed by unknown authority. So not sure where the things went wrong?

For all those suffering with the CA Certs issue and the DNS resolution refer tot hese two reported issues:
rancher/rancher#23441
rancher/rancher#16454 (comment)