govcloud-cluster-install.md

This the procedure for generating ignition configs for govcloud using th 4.3 installer (which actively thwarts attempts to do this)

Create install-config.yaml

apiVersion: v1
baseDomain: sjennings.me
# disconnected install requires local mirror
# oc adm release mirror
imageContentSources:
- mirrors:
  - registry.ocp.sjennings.me/ocp/release
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - registry.ocp.sjennings.me/ocp/release
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
compute:
- name: worker
  replicas: 2
  platform:
    aws:
      type: m4.xlarge
      zones:
      - us-west-1a
controlPlane:
  name: master
  replicas: 3
  platform:
    aws:
      type: m4.xlarge
      zones:
      - us-west-1a
metadata:
  name: ocp
platform:
  aws:
    amiID: ami-e9426288 <-- imported AMI
    region: us-west-1 <-- we have to do this because installer immediately fails for us-gov-west-1
pullSecret: 'insert-pull-secret-here'
sshKey: 'ssh-rsa AAAA...'

Generate manifests

oc adm release extract --command=openshift-install quay.io/openshift-release-dev/ocp-release:4.3.0-rc.3-x86_64
mkdir govcloud
# copy install-config.yaml from above to govcloud directory
cp install-config.yaml govcloud
# set AWS commerical admin credentials
export AWS_PROFILE=openshift-dev
./openshift-install create manifests --dir=govcloud
cd govcloud
# observe "manifests" and "openshift" directories

Modify manifests

• find/replace us-west-1 with us-gov-west-1 in all yaml files
• (optional) deuniquify infrastructure id by finding unique infrastructure name (status.infrastructureName in manifests/cluster-infrastructure-02-config.yml) and find/replace with just ocp so you don't have to recreate all the other UPI resources each time you recreate the cluster
• rm openshift/99_cloud-creds-secret.yaml 99_role-cloud-creds-secret-reader.yaml

Create ingress and machine-api IAM users

Take note of the access id and keys for the created users. Those will need to be base64 encoded and put in secrets in the next step.

ingress policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "elasticloadbalancing:DescribeLoadBalancers",
                "route53:ListHostedZones",
                "route53:ChangeResourceRecordSets",
                "tag:GetResources"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

machine-api policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeImages",
                "ec2:DescribeInstances",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVpcs",
                "ec2:RunInstances",
                "ec2:TerminateInstances",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                "elasticloadbalancing:RegisterTargets",
                "iam:PassRole"
            ],
            "Resource": "*"
        }
    ]
}

image-registry policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:CreateBucket",
                "s3:DeleteBucket",
                "s3:PutBucketTagging",
                "s3:GetBucketTagging",
                "s3:PutBucketPublicAccessBlock",
                "s3:GetBucketPublicAccessBlock",
                "s3:PutEncryptionConfiguration",
                "s3:GetEncryptionConfiguration",
                "s3:PutLifecycleConfiguration",
                "s3:GetLifecycleConfiguration",
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:HeadBucket",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucketMultipartUploads",
                "s3:AbortMultipartUpload"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Create secrets

openshift/99_openshift-ingress-operator_cloud-credentials-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: cloud-credentials
  namespace: openshift-ingress-operator
data:
  aws_access_key_id:  <base64-encoded-access-key-id>
  aws_secret_access_key: <base64-encoded-secret-access-key>

openshift/99_openshift-machine-api_aws-cloud-credentials-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: aws-cloud-credentials
  namespace: openshift-machine-api
data:
  aws_access_key_id:  <base64-encoded-access-key-id>
  aws_secret_access_key: <base64-encoded-secret-access-key>

openshift/99_openshift-image-registry_installer-cloud-credentials-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: installer-cloud-credentials
  namespace: openshift-image-registry
data:
  aws_access_key_id:  <base64-encoded-access-key-id>
  aws_secret_access_key: <base64-encoded-secret-access-key>

Generate ignition configs

• ./openshift-install create ignition-configs --dir=govcloud

The ignition and auth files that can be used for cluster installation and access are now in govcloud/

Create EC2 and ELB Endpoints in the VPC

For a disconnected environment, the normal EC2 and ELB endpoints ({ec2|elasticloadbalancing|s3}.$region.amazonaws.com) resolve to public IPs and are not reachable. A VPC Endpoint needs to be created and attached to the subnet on which the cluster instances are attached.

Additional steps to get the cluster fully functional

Disable cloud-credential-operator

$ oc edit cm cloud-credential-operator-config
...
data:
  disabled: "true"

Manually update the apps wildcard record in the private and public zones

You will need to figure out the IP of the LB that is created as a result of the Service with type: LoadBalancer that ingress creates and update the records to point to it. In a disconnected private cluster, the IP for the private zone will need to be the IP of the LB interface on the subnet so that nodes can reach it (as opposed to the external LB name/address)

TDB disable insight-operator in disconnected cluster

TDB disable or redirect samples-operator in disconneted cluster

Hi @sjenning, how do you handle the lack of a public zone in govcloud here?