failed_deployment_recovery_openshift_4_0.12.0.md

Removing a partially provisioned OpenShift 4.x AWS cluster

When creating an OCP cluster using openshift-installer that fails before the metadata.json file is created, cleaning up can be difficult because it doesn't know what needs to be removed. Fortunately, there is a workaround:

• Configure AWS CLI

  This assumes you have configured the AWS CLI using your credentials. If you have not done this, follow the instructions

• Retrieve the cluster ID

  Double check the information in the below command to ensure it's accurate for your deployment, e.g. the AWS region.

  # if using an RHPDS/opentlc instance, your cluster name is probably "cluster-GUID" 
  aws ec2 describe-instances --region=<AWS_REGION> --filters "Name=tag:clusterid,Values=<CLUSTER_NAME>" --output table
  

• Create the metadata.json file from a template

  Replace the cluster id with your data

  cat <<EOL > metadata.json
  {"clusterName":"<CLUSTER_NAME>","clusterID":"<CLUSTER_ID>","aws":{"region":"<AWS_REGION>","identifier":[{"openshiftClusterID":"<CLUSTER_ID>"},{"kubernetes.io/cluster/<CLUSTER_NAME>":"owned"}]}}
  EOL
  

• Destroy the cluster

  openshift-installer destroy cluster
  

Generating the JSON automatically

A short bash script which will generate the JSON for you has been attached to this gist.

Usage:

./meta.sh cluster-0000 > metadata.json

meta.sh

#! /usr/bin/bash

CLUSTER_NAME=$1
AWS_REGION=$2

function print_help {
    echo ""
    echo "Usage: meta.sh CLUSTER_NAME AWS_REGION"
    echo "  Default region is us-east-1"
    echo ""
    echo "  Example using default region (us-east-1): "
    echo "    meta.sh cluster-0000"
    echo "  Example providing region: "
    echo "    meta.sh cluster-0000 us-west-1"
    echo ""
}

# check for jq
JQ=`which jq`

if [ "$?" -ne "0" ]; then
    echo "ERROR: Unable to find jq binary in path"
    exit -1
fi

# check for options
if [ "$#" -ne 1 ] && [ "$#" -ne 2 ]; then
    echo "ERROR: You must supply one or two arguments"
    print_help
    exit -1
fi

# set the default AWS region
if [ -z "$AWS_REGION" ]; then
    AWS_REGION="us-east-1"
fi

# check for aws credentials
AWS_IDENT=`aws sts get-caller-identity`

if [ "$?" -ne "0" ]; then
    echo "ERROR: You do not appear to have valid AWS credentials stored"
    exit -1
fi

# retrieve the AWS instance data and cluster ID
CLUSTER_ID=$(
    aws ec2 describe-instances --region=${AWS_REGION} --filters "Name=tag:clusterid,Values=${CLUSTER_NAME}" |
    jq -r '.Reservations[0].Instances[0].Tags | .[] | select (.Key == "openshiftClusterID") | .Value' 2>/dev/null
)

if [ -z "$CLUSTER_ID" ]; then
    echo "ERROR: No cluster ID was found for ${CLUSTER_NAME}"
    exit -1
fi

# verify the cluster ID by getting the cluster name
NAME_CHECK=$(
    aws ec2 describe-instances --region=${AWS_REGION} --filters "Name=tag:openshiftClusterID,Values=${CLUSTER_ID}" |
    jq -r '.Reservations[0].Instances[0].Tags | .[] | select (.Key == "clusterid") | .Value' 2>/dev/null
)

if [ $CLUSTER_NAME != $NAME_CHECK ]; then
    echo "ERROR: The supplied name (${CLUSTER_NAME}) was not returned when searching by Cluster ID.  Value ${NAME_CHECK} was returned"
    exit -1
fi

# print the json for the metadata file
cat <<EOL
{
    "clusterName":"${CLUSTER_NAME}",
    "clusterID":"${CLUSTER_ID}",
    "aws":{
        "region":"${AWS_REGION}",
        "identifier":[
            {"openshiftClusterID":"${CLUSTER_ID}"},
            {"kubernetes.io/cluster/${CLUSTER_NAME}":"owned"}
        ]
    }
}
EOL