cloudbuild-private-cluster

cloudbuild-private-cluster.yaml

  - name: gcr.io/cloud-builders/docker
    env:
      - CLOUDSDK_COMPUTE_ZONE=us-central1
      - CLOUDSDK_CORE_PROJECT=PROJECT-NAME
    args:
      - '-c'
      - >-
        docker run -d --net cloudbuild -p 8080:8080 --name kubectl-proxy
        gcr.io/cloud-builders/gcloud compute start-iap-tunnel kubectl-bastion
        8080 --local-host-port 0.0.0.0:8080 --zone us-central1-a --project
        PROJECT-NAME && sleep 5
    id: connect-kubectl-bastion
    entrypoint: sh
  - name: gcr.io/cloud-builders/kubectl
    env:
      - CLOUDSDK_COMPUTE_ZONE=us-central1
      - CLOUDSDK_CONTAINER_CLUSTER=CLUSTER-NAME
      - CLOUDSDK_CORE_PROJECT=PROJECT-NAME
    args:
      - '-c'
      - 'HTTPS_PROXY=socks5://127.0.0.1:8080 kubectl apply -f yamls/'
    id: deploy-to-cluster
    entrypoint: sh

Received the below error, The connection to the server localhost:8080 was refused - did you specify the right host or port?

Interesting, and can you show the config for your Dante server? I believe it's running on 1080 by default, so if that's the case you'd want to use

start-iap-tunnel kubectl-bastion 1080 --local-host-port 0.0.0.0:8080

(the local port can remain the same).

Also you might want to try

HTTPS_PROXY=socks5h://127.0.0.1:8080

in case socks5:// doesn't work.

Below is the socks5 proxy server configuration

===

logoutput: syslog

internal: 0.0.0.0 port=8080

external: ens4

socksmethod: none

clientmethod: none

client pass {
from: 0.0.0.0/0 to: 0.0.0.0/0
log: connect disconnect error
}

socks pass {
from: 0.0.0.0/0 to: 0.0.0.0/0
log: connect disconnect error
}

I tried with socks5h too, it didn't work, .

I was able to make it work in cloud shell, but still i am unable to figure why it isn't working from cloud build.

Hmm, not sure why tbh, here's my config

logoutput: stderr
internal: 0.0.0.0 port = 1080
external: eth0
socksmethod: none
clientmethod: none

client pass {
  from: 0.0.0.0/0 to: 0.0.0.0/0
  log: error
}

socks pass {
  protocol: tcp
  from: 0.0.0.0/0 to: 0.0.0.0/0 port = 443
  log: error disconnect
}

socks block {
  from: 0.0.0.0/0 to: 0.0.0.0/0
  log: error connect
}

and here's how I've used it:

options:
  env:
  - BASTION_INSTANCE=gke-bastion
  - BASTION_ZONE=us-central1-a
  - BASTION_PORT=1080
  - PROXY_HOST=kubectl-proxy
  - PROXY_PORT=8080

- id: kubectl-proxy
  name: gcr.io/cloud-builders/docker
  entrypoint: sh
  args:
  - -c
  - docker run -d --net cloudbuild --name "$${PROXY_HOST}" gcr.io/cloud-builders/gcloud compute start-iap-tunnel
      "${BASTION_INSTANCE}" "${BASTION_PORT}" --local-host-port "0.0.0.0:$${PROXY_PORT}" --zone "${BASTION_ZONE}" &&
    sleep 5

- id: setup-k8s
  name: gcr.io/cloud-builders/kubectl
  entrypoint: sh
  args:
  - -c
  - HTTPS_PROXY="socks5://$${PROXY_HOST}:$${PROXY_PORT}" kubectl apply -f config.yml

I think it didn't work for you because you've used 127.0.0.1 instead of kubectl-proxy host - that name (parameterized as ${PROXY_HOST} in my config) is the name of the IAP tunnel container, so it should be the same in HTTPS_PROXY.

Thank you , I will try the same, Can you please let me know if you had used any substitutions in the build template.

@rajathithan you can use substitutions or not, up to you! My example above just uses env vars, but it doesn't matter that much! You can even hard-code the values (name of the proxy container and its local and remote ports), as long as they're consistent.

@dinvlad - I tried with both options env variables and with hard coded values, but i am still receiving the same error

"Already have image (with digest): gcr.io/cloud-builders/kubectl
The connection to the server localhost:8080 was refused - did you specify the right host or port?",

I also noticed that you had missed a double dollar sign when specifying the bastion host variab;e, I just want to let you know about it.

After trying numerous times , i had finally given up, But thanks for lending your help on this

  - name: gcr.io/cloud-builders/docker
    env:
      - CLOUDSDK_COMPUTE_ZONE=us-central1-a
      - CLOUDSDK_CORE_PROJECT=PROJECT-NAME
    args:
      - '-c'
      - >-
        docker run -d --net cloudbuild --name "$${PROXY_HOST}"
        gcr.io/cloud-builders/gcloud compute start-iap-tunnel
        "$${BASTION_INSTANCE}" "$${BASTION_PORT}" --local-host-port
        "0.0.0.0:$${PROXY_PORT}" --zone "us-central1-a"
        --project=PROJECT-NAME && sleep 5
    id: kubectl-proxy
    entrypoint: sh
  - name: gcr.io/cloud-builders/kubectl
    args:
      - '-c'
      - >-
        HTTPS_PROXY="socks5://$${PROXY_HOST}:$${PROXY_PORT}" kubectl apply -f
        yamls/
    id: setup-k8s
    entrypoint: sh


options:
  env:
    - BASTION_INSTANCE=kubectl-bastion
    - BASTION_ZONE=us-central1-a
    - BASTION_PORT=1080
    - PROXY_HOST=kubectlproxy
    - PROXY_PORT=8080

I even tried with my one (gcloud, kubectl,docker container)

- name: 'gcr.io/PROJECT-NAME/ubuntu-gkd:0.0.1'
   env:
     - CLOUDSDK_COMPUTE_ZONE=us-central1-a
     - CLOUDSDK_CORE_PROJECT=PROJECT-NAME
   args:
     - '-c'
     - >-
       docker network list && docker run -d --net cloudbuild -p 8080:8080
       --name "$${PROXY_HOST}" gcloud compute
       start-iap-tunnel "$${BASTION_INSTANCE}" "$${BASTION_PORT}"
       --local-host-port "0.0.0.0:$${PROXY_PORT}" --zone "us-central1-a"
       --project=PROJECT-NAME && sleep 5 && docker container ls |
       tail -n +2 | awk '{print $1}' | while read line; do echo "$line";
       done|xargs -n1 sh -c 'docker logs $1' sh 
   id: kubectl-proxy
   entrypoint: sh
 - name:  'gcr.io/PROJECT-NAME/ubuntu-gkd:0.0.1'
   env:
     - CLOUDSDK_COMPUTE_REGION=us-central1
     - CLOUDSDK_CONTAINER_CLUSTER=dataint-dev
     - CLOUDSDK_CORE_PROJECT=PROJECT-NAME
   args:
     - '-c'
     - >-
       HTTPS_PROXY="socks5://$${PROXY_HOST}:$${PROXY_PORT}" kubectl apply -f
       yamls/
   id: setup-k8s
   entrypoint: sh

@rajathithan sorry to hear that - that being said, this solution is not ideal in any case, since as I mentioned in the SO answer, it suffers from the concurrency/timeout issue once it's working. We've been using direct connection from Cloud Build since then (by dynamically adding its IP to authorized networks for the cluster), and this has worked very reliably.

Hi rajathithan,
Did you find a solution to this ?

No , I dont have a solution for this.

I found out that the problem is in the step 1.
When it tries to listen it doesnt output in the container "Listening on port 8080" (The bastion has the port 1080 open).
And the second thing that is weird, when you change the port to 8282 for example, kubectl keeps trying to listen on port 8080 and i didn't find anyway to make it work.

And obviously when i tried in command line, it works fine and prints "lilstening on port 8080).
I don't know why it doesnt work but i have an idea, as i have a tiny proxy installed on the bastion, i will try to tunnel throw it, maybe it will work...
another thing if it remains impossible, I'm still looking for a way to ssh on the bastion and use kubectl... I will be so thankful if you found another way.