cloudbuild-private-cluster

cloudbuild-private-cluster.yaml

  - name: gcr.io/cloud-builders/docker
    env:
      - CLOUDSDK_COMPUTE_ZONE=us-central1
      - CLOUDSDK_CORE_PROJECT=PROJECT-NAME
    args:
      - '-c'
      - >-
        docker run -d --net cloudbuild -p 8080:8080 --name kubectl-proxy
        gcr.io/cloud-builders/gcloud compute start-iap-tunnel kubectl-bastion
        8080 --local-host-port 0.0.0.0:8080 --zone us-central1-a --project
        PROJECT-NAME && sleep 5
    id: connect-kubectl-bastion
    entrypoint: sh
  - name: gcr.io/cloud-builders/kubectl
    env:
      - CLOUDSDK_COMPUTE_ZONE=us-central1
      - CLOUDSDK_CONTAINER_CLUSTER=CLUSTER-NAME
      - CLOUDSDK_CORE_PROJECT=PROJECT-NAME
    args:
      - '-c'
      - 'HTTPS_PROXY=socks5://127.0.0.1:8080 kubectl apply -f yamls/'
    id: deploy-to-cluster
    entrypoint: sh

@rajathithan you can use substitutions or not, up to you! My example above just uses env vars, but it doesn't matter that much! You can even hard-code the values (name of the proxy container and its local and remote ports), as long as they're consistent.

@dinvlad - I tried with both options env variables and with hard coded values, but i am still receiving the same error

"Already have image (with digest): gcr.io/cloud-builders/kubectl
The connection to the server localhost:8080 was refused - did you specify the right host or port?",

I also noticed that you had missed a double dollar sign when specifying the bastion host variab;e, I just want to let you know about it.

After trying numerous times , i had finally given up, But thanks for lending your help on this

  - name: gcr.io/cloud-builders/docker
    env:
      - CLOUDSDK_COMPUTE_ZONE=us-central1-a
      - CLOUDSDK_CORE_PROJECT=PROJECT-NAME
    args:
      - '-c'
      - >-
        docker run -d --net cloudbuild --name "$${PROXY_HOST}"
        gcr.io/cloud-builders/gcloud compute start-iap-tunnel
        "$${BASTION_INSTANCE}" "$${BASTION_PORT}" --local-host-port
        "0.0.0.0:$${PROXY_PORT}" --zone "us-central1-a"
        --project=PROJECT-NAME && sleep 5
    id: kubectl-proxy
    entrypoint: sh
  - name: gcr.io/cloud-builders/kubectl
    args:
      - '-c'
      - >-
        HTTPS_PROXY="socks5://$${PROXY_HOST}:$${PROXY_PORT}" kubectl apply -f
        yamls/
    id: setup-k8s
    entrypoint: sh


options:
  env:
    - BASTION_INSTANCE=kubectl-bastion
    - BASTION_ZONE=us-central1-a
    - BASTION_PORT=1080
    - PROXY_HOST=kubectlproxy
    - PROXY_PORT=8080

I even tried with my one (gcloud, kubectl,docker container)

- name: 'gcr.io/PROJECT-NAME/ubuntu-gkd:0.0.1'
   env:
     - CLOUDSDK_COMPUTE_ZONE=us-central1-a
     - CLOUDSDK_CORE_PROJECT=PROJECT-NAME
   args:
     - '-c'
     - >-
       docker network list && docker run -d --net cloudbuild -p 8080:8080
       --name "$${PROXY_HOST}" gcloud compute
       start-iap-tunnel "$${BASTION_INSTANCE}" "$${BASTION_PORT}"
       --local-host-port "0.0.0.0:$${PROXY_PORT}" --zone "us-central1-a"
       --project=PROJECT-NAME && sleep 5 && docker container ls |
       tail -n +2 | awk '{print $1}' | while read line; do echo "$line";
       done|xargs -n1 sh -c 'docker logs $1' sh 
   id: kubectl-proxy
   entrypoint: sh
 - name:  'gcr.io/PROJECT-NAME/ubuntu-gkd:0.0.1'
   env:
     - CLOUDSDK_COMPUTE_REGION=us-central1
     - CLOUDSDK_CONTAINER_CLUSTER=dataint-dev
     - CLOUDSDK_CORE_PROJECT=PROJECT-NAME
   args:
     - '-c'
     - >-
       HTTPS_PROXY="socks5://$${PROXY_HOST}:$${PROXY_PORT}" kubectl apply -f
       yamls/
   id: setup-k8s
   entrypoint: sh

@rajathithan sorry to hear that - that being said, this solution is not ideal in any case, since as I mentioned in the SO answer, it suffers from the concurrency/timeout issue once it's working. We've been using direct connection from Cloud Build since then (by dynamically adding its IP to authorized networks for the cluster), and this has worked very reliably.

Hi rajathithan,
Did you find a solution to this ?

No , I dont have a solution for this.

I found out that the problem is in the step 1.
When it tries to listen it doesnt output in the container "Listening on port 8080" (The bastion has the port 1080 open).
And the second thing that is weird, when you change the port to 8282 for example, kubectl keeps trying to listen on port 8080 and i didn't find anyway to make it work.

And obviously when i tried in command line, it works fine and prints "lilstening on port 8080).
I don't know why it doesnt work but i have an idea, as i have a tiny proxy installed on the bastion, i will try to tunnel throw it, maybe it will work...
another thing if it remains impossible, I'm still looking for a way to ssh on the bastion and use kubectl... I will be so thankful if you found another way.