Skip to content

Instantly share code, notes, and snippets.

@mak3r

mak3r/Rancher_2_air-gap_install.md

Last active March 8, 2022 00:16
Show Gist options
  • Save mak3r/a8ef14e73debc9dd9a372387c29649b6 to your computer and use it in GitHub Desktop.
Save mak3r/a8ef14e73debc9dd9a372387c29649b6 to your computer and use it in GitHub Desktop.
Download ZIP
Notes and comments on installing rancher 2 in an air-gap environment
Raw
Rancher_2_air-gap_install.md

air-gap Rancher install

This is an unofficial documentation of what I did to setup the Rancher v2 air-gap HA install

Official docs are here https://rancher.com/docs/rancher/v2.x/en/installation/air-gap-high-availability/

Install pre-reqs on Ubuntu jump host

Get kubectl

sudo snap install kubectl --classic

Get RKE

curl -L -o rke https://github.com/rancher/rke/releases/download/v0.1.11/rke_linux-amd64 && chmod 700 rke

Get helm

sudo snap install helm --classic

NOTE: The --classic flag on snaps suppresses a security warning.

error: This revision of snap "helm" was published using classic confinement and thus may perform
       arbitrary system changes outside of the security sandbox that snaps are usually confined to,
       which may put your system at risk.

       If you understand and want to proceed repeat the command including --classic.

Create an isolated network (air-gapped)

  • Build machines inside that network
  • Since I dont have on-prem systems or an AWS vpn, I am creating a poor mans air gap with EC2 instances
    • all nodes ubuntu 16.04
      • 1 node - public and private ips, jump host allows ssh from my workstation to air-gapped hosts
      • 1 node - private ip only, hosting docker repository
      • 3 nodes - private ip only, accessible only from the jump host
  • I intend to re-use the jump host as the L7 LB for the purpose of this demo

Connect to jump host

ssh -i <keyfile> ubuntu@hostip

Jump host needs access to all the private hosts

Verify

Connect through the jump box to all other hosts via ssh

Prepare copy scripts

Setup a copy script for each host to simplify moving packages required onto each host.

#!/bin/bash

scp -i <keyfile> $1 ubuntu@hostip:$2

Install docker and the docker registry on the jump host

Install Docker

The install command causes dependencies to also be installed apt install ./docker-ce_17.03.2.deb Results in

Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'docker-ce' instead of './docker-ce_17.03.2.deb'
The following additional packages will be installed:
  aufs-tools cgroupfs-mount libltdl7
Suggested packages:
  mountall
The following NEW packages will be installed:
  aufs-tools cgroupfs-mount docker-ce libltdl7
0 upgraded, 4 newly installed, 0 to remove and 4 not upgraded.
Need to get 136 kB/19.3 MB of archives.
After this operation, 89.4 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Create a .deb package repository

NOTE: For a secure apt see debian documentation https://wiki.debian.org/SecureApt Download the required package dependencies to the jump host

  • apt-get install --download-only aufs-tools cgroupfs-mount libltdl7
  • download dpkg dev apt-get install dpkg-dev
  • make a local package directory mkdir $(pwd)/air-gap-pkgs
  • Copy the extra packages that were downloaded into /var/cache/apt/archives/ into the air-gap package directory
  • cp /var/cache/apt/archives/*.deb $(pwd)/air-gap-pkgs
  • pushd air-gap-pkgs
  • dpkg-scanpackages . /dev/null > $(pwd)/Packages
  • popd
  • tar -czvf air-gap-pkgs.tgz air-gap-pkgs/

copy the docker package archive to all private hosts

  • My scp copy scripts are in the sbin directory on the jump host and are called cp1.sh, etc
  • for i in 1 2 3 reg ;do sbin/cp$i.sh air-gap-pkgs.tgz '~/.';done

Connect to each private host and install docker and dependent packages

Many of the following commands will require elevated priviliges. Elevate using sudo as necessary

  • tar -xzvf air-gap-pkgs.tgz
  • create a source list for the local package repository
  • touch /etc/apt/sources.list.d/air-gap.list
  • echo "deb file://$(pwd)/air-gap-pkgs ./" > /etc/apt/sources.list.d/air-gap.list
  • Move the internet facing source list out to the way
  • mv /etc/apt/sources.list /etc/apt/sources.list.orig
  • Update apt package manager to see the changes made to sources list
  • apt-get update
  • Install docker-ce
  • apt-get install docker-ce
  • Add the logged in user (or user who will be responsible for starting the Rancher cluster) to the docker group
  • usermod -aG docker ubuntu
  • Logout and log back in to see the user can now run docker.
  • Test that docker can be run with docker ps with no elevated privileges.

Create a docker registry

Note: I had to increase the amount disk space to > 20GB for both the jump host and the air-gap registry servers. This is due to the combination of images stored in the registry and the archive of the registry which gets copied over to the air-gap registry server.

Create an archive of the docker registry image on the jump host

  • docker pull registry && docker save docker.io/registry:latest -o docker-io.registry.tar

Load the registry on the air-gap host

  • Copy the registry volume into the air-gap registry host
  • Load the registry image from the archive
  • docker load -i docker-io.registry.tar

Fill the registry on the jump host with required images to install kubernetes and rancher

The following steps are based on the rancher docs for preparing the private registry https://staging.rancher.com/docs/rancher/v2.x/en/installation/air-gap-high-availability/prepare-private-registry/

  • curl -L -O https://github.com/rancher/rancher/releases/download/v2.1.1/rancher-images.txt
  • curl -L -O https://github.com/rancher/rancher/releases/download/v2.1.1/rancher-save-images.sh
  • curl -L -O https://github.com/rancher/rancher/releases/download/v2.1.1/rancher-load-images.sh
  • chmod +x rancher-save-images.sh
  • rke config --system-images >> ./rancher-images.txt
  • For now we just initialze the client side of helm to get the cert-manager image
    • This is only necessary for self signed certificate usage (if you are letting Rancher generate the certs)
    • helm init --client-only
  • helm template ./cert-manager-<version>.tgz | grep -oP '(?<=image: ").*(?=")' >> ./rancher-images.txt
  • Cleanup the images list to remove duplicates
  • sort -u rancher-images.txt -o rancher-images.txt
  • Now we will create an archive of the images to tranfer to the actual restistry
  • ./rancher-save-images.sh --image-list ./rancher-images.txt

Copy the archive created and the load images script to the air-gap registry host

  • Start the registry
  • docker run -d -p 5000:5000 --restart=always --name registry registry:latest
    • Note: This setup has not addressed HA for the registry and uses the docker default local filesystem for storage
  • Load the images
  • ./rancher-load-images.sh --image-list ./rancher-images.txt --registry localhost:5000
  • I used ssh tunnels to make the docker registry appear as thoug it is on localhost for every server.
    • I did this by logging into the registry server and creating remote tunnels
      ssh -f -N -R 5000:localhost:5000 ubuntu@172.31.65.224
      ssh -f -N -R 5000:localhost:5000 ubuntu@172.31.69.194
      ssh -f -N -R 5000:localhost:5000 ubuntu@172.31.70.232
  • This avoids the need to setup a certificate for the registry which is not exposed to the internet but it also means that if we were to add additional hosts, we need to make sure to create the tunnels.
    • The tunnels also will need to be recreated when the machine restarts

Setup the load balancer

I decided to use nginx as the implementation is very simple.

Rancher docs for this are here https://staging.rancher.com/docs/rancher/v2.x/en/installation/ha/create-nodes-lb/nginx/

  • I re-used my jump host as the LB (this is not recommended - put the LB inside the air-gap with a secondary NIC for external access only on 80 and 443)
  • Create the nginx.conf script on the LB
  • put the script in at /etc/nginx.conf
  • Note: the default nginx container runs with root privilege
  • Bring up the nginx container
docker run -d --restart=unless-stopped \
  -p 80:80 -p 443:443 \
  -v /etc/nginx.conf:/etc/nginx/nginx.conf \
  nginx:1.15

Install Kubernetes with RKE

https://rancher.com/docs/rancher/v2.x/en/installation/air-gap-high-availability/install-kube/

Install Rancher

https://rancher.com/docs/rancher/v2.x/en/installation/air-gap-high-availability/install-rancher/

NOTES:

  1. Create a new directory and cd into it before running the helm commands
    • This makes it much more obvious which archives and versions will be used and substituted into the next set of commands.
    • For example: Using rancher generated certs, helm will download the cert-manager archive into the directory I am running the command from.
      • Now, when I run Install rancher part C -> Option A -> Step 2 I can clearly see the version of cert manager to use is on the archive in this directory
  2. After running the helm template ... commands for cert-manager and rancher, archive up the directories and ship them off to the jump host where you can complete the rancher install

Configure Rancher for the Private Registry

https://rancher.com/docs/rancher/v2.x/en/installation/air-gap-high-availability/config-rancher-for-private-reg/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment