Debug KubeVirt controllers

debug-KV.md

TL;DR

Apply this patch, run make && make cluster-up && make cluster-sync, do a port-forward of the virt-controller 40000 port and connect to the remote dlv dlv connect :40000.

Compile in debug mode

This example shows the steps for the virt-controller but they apply to any binary in KubeVirt.

Let's start by adding the debug compilation options to not optimize the binary. Edit cmd/virt-controller/BUILD.bazel as follows.

 go_binary(
     name = "virt-controller",
     embed = [":go_default_library"],
+    gc_goopts = [
+        "-N",
+        "-l",
+    ],
     static = "on",
     visibility = ["//visibility:public"],
     x_defs = version_x_defs(),
 )

Build delve image

KubeVirt uses golang 1.13.14 so we should use dlv 1.3.2 which is the one supporting Go 1.13.14.

FROM golang:1.13.14 as builder

RUN wget https://github.com/go-delve/delve/archive/v1.3.2.tar.gz && \
    tar xf v1.3.2.tar.gz && \
    cd delve-1.3.2/cmd/dlv && \
    go install

FROM gcr.io/distroless/base
COPY --from=builder /go/bin/dlv /usr/bin/dlv

You can build/publish this image yourself or just use the one I published at quay.io/acardace/dlv:1.3.2.

Delve image in bazel

Now we need to actually consume this image in bazel and use it as the base for the virt-controller image which will run in the cluster.

In order to do that first we need to pull the image, so add this to the WORKSPACE file at the root of the project:

container_pull(
    name = "dlv",
    registry = "quay.io",
    repository = "acardace/dlv",
    tag = "1.3.2",
)

Now we can use this image to create a new one that will run virt-controller under dlv, in cmd/virt-controller/BUILD.bazel add the following:

container_image(
    name = "virt-controller-image-debug",
    base = "@dlv//image",
    directory = "/usr/bin/",
    files = [":virt-controller"],
    ports = ["40000"],
    user = "1001",
    visibility = ["//visibility:public"],
)

To actually build and push this new image to the k8s cluster created by make cluster-up we need a few more changes.

In BUILD.bazel add:

container_push(
    name = "push-virt-controller-debug",
    format = "Docker",
    image = "//cmd/virt-controller:virt-controller-image-debug",
    registry = "$(container_prefix)",
    repository = "$(image_prefix)virt-controller-debug",
    tag = "$(container_tag)",
)

In hack/bazel-build-images.sh:

-    //cmd/virt-controller:virt-controller-image //cmd/virt-handler:virt-handler-image //cmd/virt-launcher:virt-launcher-image //tests:conformance_image
+    //cmd/virt-controller:virt-controller-image //cmd/virt-handler:virt-handler-image //cmd/virt-launcher:virt-launcher-image //tests:conformance_image //cmd/virt-controller:virt-controller-image-debug

In hack/bazel-push-images.sh:

-PUSH_TARGETS=(${PUSH_TARGETS:-other-images virt-operator virt-api virt-controller virt-handler virt-launcher conformance})
+PUSH_TARGETS=(${PUSH_TARGETS:-other-images virt-operator virt-api virt-controller virt-handler virt-launcher conformance virt-controller-debug})

Now the image will be served by the registry in cluster but it still not used by the virt-operator when deploying KubeVirt.

Deploy debug image

To make the virt-operator deploy our custom debug image we need to hack the source so that it deploys what we want when installing KubeVirt, luckily it's simple enough.

In pkg/virt-operator/resource/generate/components/deployments.go apply the following 2 diffs:

 func newPodTemplateSpec(podName string, imageName string, repository string, version string, productName string, productVersion string, pullPolicy corev1.PullPolicy, podAffinity *corev1.Affinity, envVars *[]corev1.EnvVar) (*corev1.PodTemplateSpec, error) {

-   version = AddVersionSeparatorPrefix(version)
+   //version = AddVersionSeparatorPrefix(version)
+   version = ":devel"

 func NewControllerDeployment(namespace string, repository string, imagePrefix string, controllerVersion string, launcherVersion string, productName string, productVersion string, pullPolicy corev1.PullPolicy, verbosity string, extraEnv map[string]string) (*appsv1.Deployment, error) {
    podAntiAffinity := newPodAntiAffinity("kubevirt.io", "kubernetes.io/hostname", metav1.LabelSelectorOpIn, []string{"virt-controller"})
-   deploymentName := "virt-controller"
+   deploymentName := "virt-controller-debug"
    imageName := fmt.Sprintf("%s%s", imagePrefix, deploymentName)
    env := operatorutil.NewEnvVarMap(extraEnv)
    deployment, err := newBaseDeployment(deploymentName, imageName, namespace, repository, controllerVersion, productName, productVersion, pullPolicy, podAntiAffinity, env)
@@ -308,7 +309,14 @@ func NewControllerDeployment(namespace string, repository string, imagePrefix st

    container := &deployment.Spec.Template.Spec.Containers[0]
    container.Command = []string{
-       "virt-controller",
+       "dlv",
+       "exec",
+       "/usr/bin/virt-controller",
+       "--listen=:40000",
+       "--headless=true",
+       "--api-version=2",
+       "--accept-multiclient",
+       "--",
        "--launcher-image",
        fmt.Sprintf("%s/%s%s%s", repository, imagePrefix, "virt-launcher", launcherVersion),
        "--port",
@@ -322,36 +330,41 @@ func NewControllerDeployment(namespace string, repository string, imagePrefix st
            Protocol:      corev1.ProtocolTCP,
            ContainerPort: 8443,
        },
-   }
-   container.LivenessProbe = &corev1.Probe{
-       FailureThreshold: 8,
-       Handler: corev1.Handler{
-           HTTPGet: &corev1.HTTPGetAction{
-               Scheme: corev1.URISchemeHTTPS,
-               Port: intstr.IntOrString{
-                   Type:   intstr.Int,
-                   IntVal: 8443,
-               },
-               Path: "/healthz",
-           },
-       },
-       InitialDelaySeconds: 15,
-       TimeoutSeconds:      10,
-   }
-   container.ReadinessProbe = &corev1.Probe{
-       Handler: corev1.Handler{
-           HTTPGet: &corev1.HTTPGetAction{
-               Scheme: corev1.URISchemeHTTPS,
-               Port: intstr.IntOrString{
-                   Type:   intstr.Int,
-                   IntVal: 8443,
-               },
-               Path: "/leader",
-           },
+       {
+           Name:          "dlv",
+           Protocol:      corev1.ProtocolTCP,
+           ContainerPort: 40000,
        },
-       InitialDelaySeconds: 15,
-       TimeoutSeconds:      10,
    }

This will actually make the virt-operator deploy the debug image which will run the virt-controller under dlv.

I also removed the readiness and the liveness probe so that k8s will not continuosly restart the pod as when started it will hang until a remote debug session is initiated.

Connect to the remote dlv

Finally after we deployed our new debug cluster we can just port-forward the virt-controller 4000 port where the dlv instance is listening and start debugging interactively the controller.

$ kubectl -n kubevirt port-forward virt-controller-debug-7f69fd859d-kx2wb 40000:40000
Forwarding from 127.0.0.1:40000 -> 40000
Forwarding from [::1]:40000 -> 40000

$ dlv connect :40000
Type 'help' for list of commands.
(dlv) b main.main
Breakpoint 1 set at 0x1a9a63f for main.main() cmd/virt-controller/virt-controller.go:29
(dlv) c
> main.main() cmd/virt-controller/virt-controller.go:29 (hits goroutine(1):1 total:1) (PC: 0x1a9a63f)
    24:         _ "kubevirt.io/kubevirt/pkg/monitoring/reflector/prometheus" // import for prometheus metrics
    25:         _ "kubevirt.io/kubevirt/pkg/monitoring/workqueue/prometheus" // import for prometheus metrics
    26:         "kubevirt.io/kubevirt/pkg/virt-controller/watch"
    27: )
    28:
=>  29: func main() {
    30:         watch.Execute()
    31: }