How to debug go binaries running in a ProwJob locally

debug.md

How to debug go binaries running in a ProwJob locally

(in this case debug conversion-verifier running in a CAPG verify job)

Starting the ProwJob:

TEST_INFRA=${HOME}/code/src/k8s.io/test-infra
export CONFIG_PATH=${TEST_INFRA}/config/prow/config.yaml
export JOB_CONFIG_PATH=${TEST_INFRA}/config/jobs/kubernetes-sigs/cluster-api-provider-gcp/cluster-api-provider-gcp-presubmits-main.yaml

# Modify ProwJob in test-infra YAML to sleep, e.g.
#        command:
#        - "runner.sh"
#        - "sleep"
#        - "36000"

# Start ProwJob
${TEST_INFRA}/prow/pj-on-kind.sh pull-cluster-api-provider-gcp-verify

Build & copy binaries into ProwJob Pod:

cd ${HOME}/code/src/sigs.k8s.io/cluster-api

# Copy conversion-verifier with debug symbols
cd ./hack/tools
go build -gcflags "all=-N -l"  ./conversion-verifier/main.go
k cp ./main a8af0b57-f93e-11ec-97e7-0242ac110003:/tmp/main -c test

# Copy dlv
k cp $(which dlv) a8af0b57-f93e-11ec-97e7-0242ac110003:/tmp/dlv -c test

Start binary with dlv in the ProwJob Pod:

k exec -it a8af0b57-f93e-11ec-97e7-0242ac110003 /bin/bash
=> /tmp/dlv exec --headless --listen=:20000 --api-version=2 --accept-multiclient /tmp/main

Port-forward debug port:

k port-forward a8af0b57-f93e-11ec-97e7-0242ac110003 20000

Connect with Intellij with a Go remote debug configuration to localhost:20000