Experiment on Google Cloud with Cloud Run, Cloud NAT, Private Google Access, and Secure Web Proxy with NAT only for external requests

cloud-run-nat-serial.sh

#!/usr/bin/env bash

#####################################################################
# REFERENCES
# - https://cloud.google.com/sdk/gcloud/reference/compute/networks/create
# - https://cloud.google.com/sdk/gcloud/reference/compute/networks/subnets/create
# - https://cloud.google.com/vpc/docs/configure-private-google-access
# - https://cloud.google.com/network-connectivity/docs/router/how-to/create-router-vpc-network#gcloud
# - https://cloud.google.com/nat/docs/set-up-manage-network-address-translation
# - https://cloud.google.com/nat/docs/using-nat-rules
# - https://www.youtube.com/watch?v=5qsGkfAsomM
# - https://cloud.google.com/nat/docs/using-nat-rules#list_all_nat_rules_in_a_nat_gateway
# - https://cloud.google.com/secure-web-proxy/docs/cel-matcher-language-reference
# - https://pkg.go.dev/net/http
# - https://cloud.google.com/blog/products/serverless/announcing-direct-vpc-egress-for-cloud-run
# - https://cloud.google.com/run/docs/configuring/vpc-direct-vpc#gcloud
# - https://cloud.google.com/run/docs/configuring/static-outbound-ip
#####################################################################

export PROJECT_ID=$(gcloud config get-value project)
export PROJECT_USER=$(gcloud config get-value core/account) # set current user
export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")
export IDNS=${PROJECT_ID}.svc.id.goog # workflow identity domain

export GCP_REGION="us-central1" # CHANGEME (OPT)
export GCP_ZONE="us-central1-a" # CHANGEME (OPT)
export NETWORK_NAME="default"

# enable apis
gcloud services enable compute.googleapis.com \
    storage.googleapis.com \
    run.googleapis.com \
    cloudbuild.googleapis.com \
    artifactregistry.googleapis.com \
    dns.googleapis.com \
    vpcaccess.googleapis.com \
    certificatemanager.googleapis.com \
    networksecurity.googleapis.com \
    networkservices.googleapis.com

# configure gcloud sdk
gcloud config set compute/region $GCP_REGION
gcloud config set compute/zone $GCP_ZONE


############################################################
# Networking
############################################################
export NETWORK_NAME="dev-10-0-50-0"
export SUBNET_SVC_NAME="services"
export SUBNET_SVC_RANGE="10.0.56.0/21"          # 10.0.56.0 - 10.0.63.255 (2,048)
export SUBNET_PROXY_NAME="secure-proxy"
export SUBNET_PROXY_RANGE="192.168.0.0/23"
export AUTHORIZATION_NAME="cert-auth-dns"
export DOMAIN_NAME="proxy.msparr.com"       # CHANGEME
export CERTIFICATE_NAME="proxy-cert"
export DNS_ZONE="private-zone"
export DNS_ZONE2="run-zone"
export DNS_TTL="300"
export IP_ADDRESS1_NAME="nat-static-ip-1"
export IP_ADDRESS2_NAME="nat-static-ip-2"
export IP_ADDRESS3_NAME="nat-static-ip-3"
export IP_ADDRESS4_NAME="nat-static-ip-4"
export IP_ADDRESS5_NAME="nat-static-ip-5"
export IP_ADDRESS6_NAME="nat-static-ip-6"
export NAT_ROUTER="nat-router-1"
export NAT_GATEWAY="nat-gw-1"
export ASN_NUMBER="65155"

# create network
gcloud compute networks create $NETWORK_NAME \
    --subnet-mode=custom \
    --bgp-routing-mode=global

# create proxy subnet
gcloud compute networks subnets create $SUBNET_PROXY_NAME \
    --purpose=REGIONAL_MANAGED_PROXY \
    --role=ACTIVE \
    --region=$GCP_REGION \
    --network=$NETWORK_NAME \
    --range=$SUBNET_PROXY_RANGE \
    --enable-private-ip-google-access

# create subnet for cloud run services
gcloud compute networks subnets create $SUBNET_SVC_NAME \
    --network=$NETWORK_NAME \
    --range=$SUBNET_SVC_RANGE \
    --region=$GCP_REGION \
    --enable-private-ip-google-access \
    --enable-flow-logs

# create ssh firewall rule
gcloud compute firewall-rules create allow-ssh \
    --direction=INGRESS \
    --priority=1000 \
    --network=$NETWORK_NAME \
    --action=ALLOW \
    --rules=tcp:22 \
    --source-ranges=0.0.0.0/0

# create public dns authorization for regional managed cert
gcloud certificate-manager dns-authorizations create $AUTHORIZATION_NAME \
    --domain="$DOMAIN_NAME" \
    --type=PER_PROJECT_RECORD \
    --location=$GCP_REGION

# describe dns auth to get CNAME for updating records
gcloud certificate-manager dns-authorizations describe $AUTHORIZATION_NAME \
    --location=$GCP_REGION

# update your DNS records to add CNAME as this example:
# - cname: _acme-challenge_e3k6womt3p3pu7d4.proxy.msparr.com
# - data: 903d9e2d-0782-404b-92e6-dc5a41ce54a4.1.us-central1.authorize.certificatemanager.goog.

# create regional managed certificate
gcloud certificate-manager certificates create $CERTIFICATE_NAME \
   --domains=$DOMAIN_NAME \
   --dns-authorizations=$AUTHORIZATION_NAME \
   --location=$GCP_REGION

# create private zones
gcloud dns managed-zones create $DNS_ZONE \
    --description="internal zone" \
    --dns-name="googleapis.com" \
    --networks=$NETWORK_NAME \
    --labels="purpose=demo" \
    --visibility=private
gcloud dns managed-zones create $DNS_ZONE2 \
    --description="internal zone" \
    --dns-name="run.app" \
    --networks=$NETWORK_NAME \
    --labels="purpose=demo" \
    --visibility=private

# configure private api dns
gcloud dns record-sets transaction start --zone=$DNS_ZONE
gcloud dns record-sets transaction add 199.36.153.8 199.36.153.9 199.36.153.10 199.36.153.11 \
    --name="private.googleapis.com" --ttl=$DNS_TTL --type="A" --zone=$DNS_ZONE
gcloud dns record-sets transaction add "private.googleapis.com." \
    --zone=$DNS_ZONE --name="*.googleapis.com" --type="CNAME" --ttl=$DNS_TTL 
gcloud dns record-sets transaction execute --zone=$DNS_ZONE

# configure cloud run dns
gcloud dns record-sets transaction start --zone=$DNS_ZONE2
gcloud dns record-sets transaction add 199.36.153.8 199.36.153.9 199.36.153.10 199.36.153.11 \
    --name="run.app" --ttl=$DNS_TTL --type="A" --zone=$DNS_ZONE2
gcloud dns record-sets transaction add "run.app." \
    --zone=$DNS_ZONE2 --name="*.run.app" --type="CNAME" --ttl=$DNS_TTL 
gcloud dns record-sets transaction execute --zone=$DNS_ZONE2

# create static IPs for allocating to NAT
gcloud compute addresses create $IP_ADDRESS1_NAME --region $GCP_REGION
gcloud compute addresses create $IP_ADDRESS2_NAME --region $GCP_REGION
gcloud compute addresses create $IP_ADDRESS3_NAME --region $GCP_REGION
gcloud compute addresses create $IP_ADDRESS4_NAME --region $GCP_REGION
gcloud compute addresses create $IP_ADDRESS5_NAME --region $GCP_REGION
gcloud compute addresses create $IP_ADDRESS6_NAME --region $GCP_REGION
export IP_ADDRESS1=$(gcloud compute addresses describe $IP_ADDRESS1_NAME \
    --format="value(address)" --region=$GCP_REGION)
export IP_ADDRESS2=$(gcloud compute addresses describe $IP_ADDRESS2_NAME \
    --format="value(address)" --region=$GCP_REGION)
export IP_ADDRESS3=$(gcloud compute addresses describe $IP_ADDRESS3_NAME \
    --format="value(address)" --region=$GCP_REGION)
export IP_ADDRESS4=$(gcloud compute addresses describe $IP_ADDRESS4_NAME \
    --format="value(address)" --region=$GCP_REGION)
export IP_ADDRESS5=$(gcloud compute addresses describe $IP_ADDRESS5_NAME \
    --format="value(address)" --region=$GCP_REGION)
export IP_ADDRESS6=$(gcloud compute addresses describe $IP_ADDRESS6_NAME \
    --format="value(address)" --region=$GCP_REGION)


# # optional (manually-created NAT)

# # create cloud router
# gcloud compute routers create $NAT_ROUTER \
#     --project=$PROJECT_ID \
#     --network=$NETWORK_NAME \
#     --asn=$ASN_NUMBER \
#     --region=$GCP_REGION

# # create nat gateway
# gcloud compute routers nats create $NAT_GATEWAY \
#     --router=$NAT_ROUTER \
#     --region=$GCP_REGION \
#     --enable-logging \
#     --nat-all-subnet-ip-ranges \
#     --nat-external-ip-pool=$IP_ADDRESS1_NAME,$IP_ADDRESS2_NAME


############################################################
# Secure Web Proxy
############################################################
export POLICY_NAME="policy1"
export POLICY_FILE_NAME="policy.yaml"
export RULE_NAME="allow-httpbin-org"
export RULE_FILE_NAME="rule.yaml"
export URL_LIST_NAME="allowed-ext-url-list"
export URL_LIST_FILE="urllist.yaml"
export URL_LIST_PATH="projects/$PROJECT_ID/locations/$GCP_REGION/urlLists/$URL_LIST_NAME"
export GATEWAY_NAME="swp1"
export GATEWAY_FILE_NAME="gateway.yaml"
export GATEWAY_PROXY_IP="10.0.63.200"
export GATEWAY_ADDRESSES="[$GATEWAY_PROXY_IP]"
export GATEWAY_PROXY_PORT="443"
export GATEWAY_PORTS="[$GATEWAY_PROXY_PORT]"

# create policy file (skipped TLS inspection / CA setup for simplicity)
cat > $POLICY_FILE_NAME << EOF
description: basic Secure Web Proxy policy
name: projects/$PROJECT_ID/locations/$GCP_REGION/gatewaySecurityPolicies/$POLICY_NAME
EOF

# create the swp policy
gcloud network-security gateway-security-policies import $POLICY_NAME \
    --source=$POLICY_FILE_NAME \
    --location=$GCP_REGION

# create the url list config file of allowed external domains
cat > $URL_LIST_FILE << EOF
name: $URL_LIST_PATH
values:
  - httpbin.org
EOF

# create url list
gcloud network-security url-lists import $URL_LIST_NAME \
    --location=$GCP_REGION \
    --project=$PROJECT_ID \
    --source=$URL_LIST_FILE

# create swp rules file (skipped TLS inspection / CA setup for simplicity)
cat > $RULE_FILE_NAME << EOF
name: projects/$PROJECT_ID/locations/$GCP_REGION/gatewaySecurityPolicies/$POLICY_NAME/rules/$RULE_NAME
description: Allow httpbin.org
enabled: true
priority: 1
basicProfile: ALLOW
sessionMatcher: inUrlList(host(), "$URL_LIST_PATH")
EOF

# create the swp rule
gcloud network-security gateway-security-policies rules import $RULE_NAME \
    --source=$RULE_FILE_NAME \
    --location=$GCP_REGION \
    --gateway-security-policy=$POLICY_NAME

# create the gateway config file
cat > $GATEWAY_FILE_NAME << EOF
name: projects/$PROJECT_ID/locations/$GCP_REGION/gateways/$GATEWAY_NAME
type: SECURE_WEB_GATEWAY
addresses: $GATEWAY_ADDRESSES
ports: $GATEWAY_PORTS
certificateUrls: ["projects/$PROJECT_ID/locations/$GCP_REGION/certificates/$CERTIFICATE_NAME"]
gatewaySecurityPolicy: projects/$PROJECT_ID/locations/$GCP_REGION/gatewaySecurityPolicies/$POLICY_NAME
network: projects/$PROJECT_ID/global/networks/$NETWORK_NAME
subnetwork: projects/$PROJECT_ID/regions/$GCP_REGION/subnetworks/$SUBNET_SVC_NAME
scope: samplescope
EOF

# create the gateway
gcloud network-services gateways import $GATEWAY_NAME \
    --source=$GATEWAY_FILE_NAME \
    --location=$GCP_REGION

# update NAT gateway (static IP, specific subnet, min ports/vm)
export PROXY_NAT_NAME="swg-autogen-nat"
export PROXY_ROUTER_NAME=$(gcloud compute routers list --filter="name:swg-*" --format="value(name)")

gcloud compute routers nats update swg-autogen-nat \
    --router=$PROXY_ROUTER_NAME \
    --region=$GCP_REGION \
    --nat-external-ip-pool="$IP_ADDRESS1_NAME,$IP_ADDRESS2_NAME" \
    --nat-custom-subnet-ip-ranges=$SUBNET_SVC_NAME \
    --enable-dynamic-port-allocation \
    --min-ports-per-vm=64 \
    --max-ports-per-vm=16384 \
    --tcp-time-wait-timeout=30 \
    --enable-logging \
    --log-filter=ALL

# create rules for nat gw for httpbin.org
export NAT_RULE_NUMBER=1000

echo "Creating NAT rule [$NAT_RULE_NUMBER] for $PROXY_NAT_NAME"

gcloud compute routers nats rules create $NAT_RULE_NUMBER \
    --router=$PROXY_ROUTER_NAME \
    --nat=$PROXY_NAT_NAME \
    --match="destination.ip == '54.242.212.250'" \
    --source-nat-active-ips="$IP_ADDRESS6_NAME" \
    --region=$GCP_REGION

# gcloud compute routers nats rules delete $NAT_RULE_NUMBER \
#     --router=$PROXY_ROUTER_NAME \
#     --nat=$PROXY_NAT_NAME


############################################################
# Test service
# - Cloud Run app that posts its payload to configurable URL
############################################################
export SVC_NAME_BASE="grapevine"

# create sample service
mkdir -p $SVC_NAME_BASE

# create cloud ignore file
cat > .gcloudignore << EOF
.git
*.sh
EOF

# create mod file
cat > $SVC_NAME_BASE/go.mod << EOF
module github.com/mikesparr/$SVC_NAME_BASE
go 1.20
EOF

# create sample app
cat > $SVC_NAME_BASE/main.go << EOF
package main
import (
	"fmt"
	"net/http"
    "crypto/tls"
	"os"
)

func main() {
    port := os.Getenv("PORT")
    service := os.Getenv("K_SERVICE")
    target := os.Getenv("TARGET")

	if port == "" {
		panic("PORT required")
	}
	if service == "" {
		panic("K_SERVICE required")
	}
	if target == "" {
		panic("TARGET required")
	}

	transport := http.Transport{
		Proxy:           http.ProxyFromEnvironment,
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}

	client := &http.Client{Transport: &transport}

	http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
        if req.Method == http.MethodPost {
            fmt.Println("POST received in:", service)

            resp, err := client.Post(target, req.Header.Get("Content-Type"), req.Body)
            if err != nil {
                fmt.Printf("POST failed to: %s with error: %s\n", target, err.Error())
                http.Error(w, "Error sending POST: " + err.Error(), http.StatusInternalServerError)
                return
            }

            req.Body.Close()
            resp.Body.Close()

            fmt.Println("POST done to:", target)
        } else {
            http.Error(w, "Invalid request method", http.StatusMethodNotAllowed)
        }
	})
	fmt.Printf("Listening on :%s\n", port)
	if err := http.ListenAndServe(":"+port, nil); err != nil {
		panic(err)
	}
}
EOF


############################################################
# Experiment
# - deploy N instances of test app that passes along requests
#   to configured URL and logs invocation
# - name service by ordinal index in reverse (last external)
# - start chain by POSTing to first ordinal service
# - ensure initial requests all internal and only last
#   uses Cloud NAT
############################################################
export EXTERNAL_URL="https://httpbin.org/post"
export TARGET=$EXTERNAL_URL # first service go through NAT, rest internal chain

# add IAM policy binding to allow Cloud Run access to network
gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member "serviceAccount:service-$PROJECT_NUMBER@serverless-robot-prod.iam.gserviceaccount.com" \
    --role "roles/compute.networkUser"

# deploy services in reverse order with last one that posts externally
for i in {5..1}
do
    echo "Deploying service $SVC_NAME_BASE-$i with target: $TARGET"
    gcloud beta run deploy $SVC_NAME_BASE-$i \
        --source $SVC_NAME_BASE \
        --region=$GCP_REGION \
        --network=$NETWORK_NAME \
        --subnet=$SUBNET_SVC_NAME \
        --ingress=all \
        --vpc-egress=all-traffic \
        --allow-unauthenticated \
        --set-env-vars "TARGET=$TARGET"
    TARGET=$(gcloud run services describe $SVC_NAME_BASE-$i --region $GCP_REGION --format="value(status.url)")
    echo "Updated next target to: $TARGET"
done

# update service ENV var for external with http proxy
gcloud beta run services update $SVC_NAME_BASE-5 \
    --update-env-vars=HTTP_PROXY="https://$GATEWAY_PROXY_IP:$GATEWAY_PROXY_PORT" \
    --region=$GCP_REGION

# test invocation using last TARGET value
echo "Testing $TARGET ..."
curl -X POST -H "Content-Type: application/json" $TARGET -d '{"data": "foo"}'
echo "Done testing! Check the logs..."

# check NAT gateway logs
gcloud logging read 'resource.type=nat_gateway' \
    --limit=10 \
    --format=json

Google Cloud Run Serial Invocation Test (1)

This experiment tests whether using Direct VPC Egress on Google Cloud with Cloud Run services, and Private Google Access on the designated subnet, will result in bypassing Cloud NAT for service-to-service requests, but using Cloud NAT for external requests only.

Architecture

Setup

The code above loops from 5 to 1, first setting the TARGET env var which is destination URL for test app POST request. It then updates the variable with the Cloud Run service URL for the next target, and repeats to ordinal index -1.

Deployed

Once all instances are deployed, the last URL is the one we call with curl

Test

The experiment then invokes one or more requests to the first service URL (which will be the last TARGET set in env), to start the serial chain of requests. The first 4 requests will just call another service, ideally using only internal network, and the last request will call the external URL https://httpbin.org/post and that should use Cloud NAT.

# test invocation using last TARGET value
echo "Testing $TARGET ..."
curl -X POST -H "Content-Type: application/json" $TARGET -d '{"data": "foo"}'
echo "Done testing! Check the logs..."

Result

Still configuring as unexpected results

Updated to use Secure Web Proxy

After updating the original experiment with manually-created NAT and Router, used Secure Web Proxy and certificate from my personal domain proxy.msparr.com with CNAME for DNS verification.

• Required change to Go application code using client and Transport to declare proxy and disable TLS verification

	transport := http.Transport{
		Proxy:           http.ProxyFromEnvironment,
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}

	client := &http.Client{Transport: &transport}

• Did work and as illustrated below, request ran through Secure Web Proxy and triggered rule to allow httpbin.org

Solution

After restricting subnets to omit the Secure Web Proxy subnet range, Cloud NAT began working as expected even with a single static IP - hooray. I even tested adding additional nat rule with single IP and worked fine without errors.