This gist represents the files and process used during the Ask an OpenShift Admin livestream from Nov 10th 2021: https://www.youtube.com/watch?v=VkP2PRNanAI.

disconnected_deep_dive.md

Mirroring images

This follows the documentation for mirroring images.

1. Download the images

   • Use dryrun.sh to get the `ImageContentSourcePolicy`` needed for the disconnected cluster.

     The values used for the destination registry, which are used for the ICSP, can be arbitrary and changed on the disconnected network to represent your scenario. This is useful if the hostnames / IPs are sensitive.

   • Use mirror.sh to mirror the content to a (removable) disk.

   We also need the Docker registry image. This isn't necessary if you have a registry on the disconnected network.

   podman pull docker.io/library/registry:2
   podman save -o registry-2.tar docker.io/library/registry:2
   mv registry-2.tar $REMOVEABLE_MEDIA_PATH

   Finally, we'll want the client tools.

   wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.9.4/openshift-client-linux-4.9.4.tar.gz
   wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.9.4/openshift-install-linux-4.9.4.tar.gz
   
   mv *.tar.gz $REMOVABLE_MEDIA_PATH

2. [Optional] Stand up a disconnected registry. This is not needed if you already have a registry on the disconnected network.

   From a host on the disconnected network, create a registry instance with certificates.

   # import the registry image
   podman load -i registry-2.tar
   
   # start the registry using the certificate and providing the desired
   # storage location
   podman run -d --restart=always --name registry \
    -p 5000:5000 \
    -v $PWD/certs:/certs \
    -v /mnt/registry/data:/var/lib/registry \
    -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
    -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
   docker.io/library/registry:2
   
   curl -v https://bastion.lab.lan:5000/v2/_catalog

   If you need to generate certificates, this will work for most cases:

   # some where to hold certificates
   mkdir certs
   
   # generate a new certificate and key
   openssl req  \
    -newkey rsa:4096 \
    -nodes -sha256 \
    -keyout certs/domain.key \
    -x509 -days 365 \
    -out certs/domain.crt
   
   # add them to the local host's trust
   sudo cp registry/domain.crt /etc/pki/ca-trust/source/anchors/registry.crt
   sudo update-ca-trust extract

   The above process doesn't create sANs for the certificate, so you'll need to prepend this to the oc commands:

   GODEBUG=x509ignoreCN=0 oc ...

3. Import the images to the disconnected registry

   After bringing the data over to the disconnected network, use import.sh to import from disk to the disconnected registry.

Installing disconnected

Following from here, we're doing to do a disconnected single node OpenShift installation using static IPs.

1. Create the install-config.yaml

   We need to add the imageContentSources and additionalTrustBundle information to the install-config.yaml so that it knows to connect to the disconnected registry for images and it will do so without certificate errors. Replace the destination mirror with appropriate values for your environment.

   We also need to append the --copynetwork value to the bootstrapInPlace.installationDisk parameter. This is what persists the static IP configuration from the bootstrap phase - where we will provide network configuration via kernel parameters - to the installed RHCOS instance.

   We do not need to use a valid Red Hat pull secret here, however if your disconnected registry requires authentication, that information should be provided in the pullSecret field.

   apiVersion: v1
   baseDomain: lab.lan
   metadata:
     name: sno
   imageContentSources:
   - mirrors:
     - disconnected.registry.tld:5000/openshift/4.9.4
     source: quay.io/openshift-release-dev/ocp-release
   - mirrors:
     - disconnected.registry.tld:5000/openshift/4.9.4
     source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
   additionalTrustBundle: |
     -----BEGIN CERTIFICATE-----
     ...
     -----END CERTIFICATE-----
   networking:
     networkType: OVNKubernetes
     machineNetwork:
     - cidr: 10.0.101.0/24
   compute:
   - name: worker
     replicas: 0
   controlPlane:
     name: master
     replicas: 1
   platform:
     none: {}
   bootstrapInPlace:
     installationDisk: "/dev/sda --copynetwork"
   pullSecret: '{"auths":{"fake":{"auth":"aWQ6cGFzcwo="}}}'
   sshKey: |
     ssh-ed25519 keygoeshere

2. Follow steps 3 and 4 from the other gist

3. When booting the VM, interrupt the prompt to add static IP information

   This follows the standard methodology from the documentation.

   ip=10.0.101.60::10.0.101.1:255.255.255.0:sno-node.lab.lan:ens192:none nameserver=192.168.14.39
   

   After appending the network configuration data, boot RHCOS and allow it to proceed.

4. Monitor the install using the standard methods

   openshift-install wait-for bootstrap-complete and openshift-install wait-for install-complete will both provide status info for the installation. Additionally, SSH will work if needed.

Disconnected Operator Catalog

This follows the documentation for using OLM on a restricted network.

1. Prune the index

   # login to the registry endpoints we need
   podman login registry.redhat.io
   podman login quay.io
   
   # if you need a list of the catalog items
   # podman run -p 50051:50051 -it registry.redhat.io/redhat/redhat-operator-index:v4.8
   # grpcurl -plaintext localhost:50051 api.Registry/ListPackages > packages.out
   
   # create a pruned image
   opm index prune \
    -f registry.redhat.io/redhat/redhat-operator-index:v4.9 \
    -p cincinnati-operator \
    -t quay.io/ansulliv/cincinnati:v4.9
   
   # needs to be in an actual registry, not a local image, but doesn't need
   # to be quay to work
   podman push quay.io/ansulliv/cincinnati:v4.9

2. Mirror the content

   Doing a dry run first will provide, among other things, the amount of data which will be pulled down.

   # pull the content locally, use your pull secret for credentials
   oc adm catalog mirror --dry-run \
     -a ~/pull_secret/pull-secret.txt \
    quay.io/ansulliv/cincinnati:v4.9 \
    file:///local/index

   For example, with this scenario (just the cincinnati Operator), it will mirror approximately 34GiB of data:

   stats: shared=157 unique=1256 size=34.55GiB ratio=0.86
   

   With the dry run (optionally) done, remove the parameter to have it copy the data to a local disk location.

   oc adm catalog mirror \
     -a ~/pull_secret/pull-secret.txt \
    quay.io/ansulliv/cincinnati:v4.9 \
    file:///local/index

   Move the data to the disconnected network for the next step.

3. Import the content on the disconnected network

   oc adm catalog mirror \
    file://local/index/ansulliv/cincinnati:v4.9 \
    bastion.lab.lan:5000/cincinnati

4. Add an ImageContentSourcePolicy and CatalogSource to the disconnected cluster

   Examples for both of these can be found in the directory generated by the previous oc adm catalog mirror command, for example ./manifests-index/catalog/name/{catalogSource.yaml,imageContentSourcePolicy.yaml}

dry_run.sh

#! /usr/bin/env sh

#
# replace all values with those appropraite for your environment
#

OCP_RELEASE=4.9.4
PRODUCT_REPO='openshift-release-dev'
LOCAL_SECRET_JSON=~/pull-secret.txt
RELEASE_NAME="ocp-release"
ARCHITECTURE=x86_64
REMOVABLE_MEDIA_PATH=~/disconnected/mirror/4.9

LOCAL_REGISTRY=registry.lab.lan:5000
LOCAL_REPOSITORY=openshift/4.9.4

oc adm release mirror -a ${LOCAL_SECRET_JSON} \
 --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
 --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
 --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} \
 --dry-run

import.sh

#! /usr/bin/env sh

#
# replace all values with those appropraite for your environment
#

OCP_RELEASE=4.9.4
PRODUCT_REPO='openshift-release-dev'
RELEASE_NAME="ocp-release"
ARCHITECTURE=x86_64
REMOVABLE_MEDIA_PATH=/mnt/disconnected/4.9/images

LOCAL_REGISTRY=bastion.lab.lan:5000
LOCAL_REPOSITORY=openshift/4.9.4

oc image mirror \
 --from-dir=${REMOVABLE_MEDIA_PATH} "file://openshift/release:${OCP_RELEASE}*" \
 ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}

LOCAL_RELEASE_IMAGES_REPOSITORY='ocp4/openshift4-release-images'

oc image mirror \
  ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} \
  ${LOCAL_REGISTRY}/${LOCAL_RELEASE_IMAGES_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

mirror.sh

#! /usr/bin/env sh

#
# replace all values with those appropraite for your environment
#

OCP_RELEASE=4.9.4
PRODUCT_REPO='openshift-release-dev'
LOCAL_SECRET_JSON=~/pull-secret.txt
RELEASE_NAME="ocp-release"
ARCHITECTURE=x86_64
REMOVABLE_MEDIA_PATH=~/disconnected/mirror/4.9

oc adm release mirror -a ${LOCAL_SECRET_JSON} \
 --to-dir=${REMOVABLE_MEDIA_PATH}/images \
 quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE}