We use cert-manager in our gcp hosted openshift clusters installed with operator hub. The default installation method sets up split horizon dns which thwarts DNS01 solver. Challenges are stuck in the pending state. The real solution is to change the cert-manager deployment so that it include the arguments to use recursive name servers. Unfortuna…

cert-manager-delegation.sh

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'
set -x

GC_PROJECT="${GC_PROJECT:-my-dev-project}"
NEW_RECORD_TTL="${NEW_RECORD_TTL:-59}"

if [[ -e transaction.yaml ]]; then
  echo "WARN: Found existing transation.yaml, removing interactively"
  rm -i transaction.yaml
fi

if [[ -n ${PRIVATE_CLOUDDNS_ZONE:-} && -z ${SOURCE_DOMAIN:-} ]]; then
  SOURCE_DOMAIN=$(gcloud --project="${GC_PROJECT}" dns managed-zones list \
    --filter="NAME:${PRIVATE_CLOUDDNS_ZONE} AND VISIBILITY:private" \
    --format='value(DNS_NAME)' \
  )
fi

if [[ -z ${SOURCE_DOMAIN:-} ]]; then
  #TODO: Probably a better way to handle this than two seperate
  #variable assignements
  PRIVATE_CLOUDDNS_ZONE=$(gcloud --project="${GC_PROJECT}" dns managed-zones list \
    --filter='NAME:*-private-zone AND VISIBILITY:private' \
    --format='value(NAME)' \
  )
  SOURCE_DOMAIN=$(gcloud --project="${GC_PROJECT}" dns managed-zones list \
    --filter='NAME:*-private-zone AND VISIBILITY:private' \
    --format='value(DNS_NAME)' \
  )
fi

if [[ -z ${PRIVATE_CLOUDDNS_ZONE:-} ]]; then
  PRIVATE_CLOUDDNS_ZONE=$(gcloud --project="${GC_PROJECT}" dns managed-zones list \
    --filter="DNS_NAME:${SOURCE_DOMAIN} AND VISIBILITY:private" \
    --format='value(NAME)' \
  )
fi

if [[ -z ${CLOUDDNS_ZONE:-} ]]; then
  CLOUDDNS_ZONE=$(gcloud --project="${GC_PROJECT}" dns managed-zones list \
    --filter="DNS_NAME:${SOURCE_DOMAIN} AND VISIBILITY:public" \
    --format='value(NAME)' \
  )
fi

case "${SOURCE_DOMAIN}" in

  "dev.example.com.")
    TARGET_DOMAIN="_acme-delegation.acme.example.com."
    ;;

  "prod.example.com.")
    TARGET_DOMAIN="_acme-delegation.acme.example.com."
    ;;

  *)
    echo "FATAL: Could not match SOURCE_DOMAIN, ${SOURCE_DOMAIN}, to a TARGET_DOMAIN"
    exit 1
    ;;
esac

gcloud --project="${GC_PROJECT}" dns record-sets transaction start --zone="${CLOUDDNS_ZONE}"

#First setup public CLOUDDNS_ZONE
gcloud --project="${GC_PROJECT}" dns record-sets list --zone="${CLOUDDNS_ZONE}" \
  --filter='NAME:_acme-challenge.*' \
  --format 'value(NAME,TYPE,TTL,DATA)' \
| while read record type ttl data; do
  #Delete existing record
  gcloud --project="${GC_PROJECT}" dns record-sets transaction remove "${data}" \
    --name="${record}" \
    --ttl="${ttl}" \
    --type="${type}" \
    --zone="${CLOUDDNS_ZONE}"

  #Create CNAME (delegation) records
  gcloud --project="${GC_PROJECT}" dns record-sets transaction add "${record}${TARGET_DOMAIN}" \
    --name="${record}" \
    --ttl="${NEW_RECORD_TTL}" \
    --type='CNAME' \
    --zone="${CLOUDDNS_ZONE}"
done

gcloud --project="${GC_PROJECT}" dns record-sets transaction execute --zone="${CLOUDDNS_ZONE}"
gcloud --project="${GC_PROJECT}" dns record-sets transaction start --zone="${PRIVATE_CLOUDDNS_ZONE}"

#Second clean up private CLOUDDNS_ZONE
gcloud --project="${GC_PROJECT}" dns record-sets list \
  --zone="${PRIVATE_CLOUDDNS_ZONE}" \
  --filter='NAME:_acme-challenge.*' \
  --format='value(NAME,TYPE,TTL,DATA)' \
| while read record type ttl data; do
  #Delete existing record
  gcloud --project="${GC_PROJECT}" dns record-sets transaction remove "${data}" \
    --name="${record}" \
    --ttl="${ttl}" \
    --type="${type}" \
    --zone="${PRIVATE_CLOUDDNS_ZONE}"
done

#Third copy CNAME (delegation) records from CLOUDDNS_ZONE to PRIVATE_CLOUDDNS_ZONE
gcloud --project="${GC_PROJECT}" dns record-sets list \
  --zone="${CLOUDDNS_ZONE}" \
  --filter="NAME:_acme-challenge.* AND TYPE:CNAME AND RRDATAS:*.${TARGET_DOMAIN}" \
  --format='value(NAME,TYPE,TTL,DATA)' \
| while read record type ttl data; do
  #Create CNAME (delegation) records
  gcloud --project="${GC_PROJECT}" dns record-sets transaction add "${data}" \
    --name="${record}" \
    --ttl="${ttl}" \
    --type="${type}" \
    --zone="${PRIVATE_CLOUDDNS_ZONE}"
done

gcloud --project="${GC_PROJECT}" dns record-sets transaction execute --zone="${PRIVATE_CLOUDDNS_ZONE}"

To finish that thought: Unfortunately the operator immediately reverts that change.

This script makes a number of assumptions including that your version of acme.example.com. is hosted in the same project as the split horizon dns zones for your cluster.

You can call it like this: GC_PROJECT=my_awesome_cluster_project ./cert-manager-delegation.sh
then delete all of your challenges: oc delete challenge -A --all