Hacking trace into kata proxy

Hack_trace_into_kata_proxy.md

I needed to try and grab the golang GODEBUG gc and scheduler trace debug info, from the Kata proxy. But, how - these are enabled by the ENV and/or command line - and the proxy is run from the runtime, which is run from dockerd. Well, this might just require some patches...

In the proxy itself:

diff --git a/proxy.go b/proxy.go
index 2a51f16..4a054bb 100644
--- a/proxy.go
+++ b/proxy.go
@@ -18,6 +18,8 @@ import (
        "net/url"
        "os"
        "os/signal"
+//     "runtime"
+       "runtime/trace"
        "sync"
        "syscall"
        "time"
@@ -450,5 +452,18 @@ func realMain() {

 func main() {
        defer handlePanic()
+
+       if _, err := os.Stat("/tmp/trace_proxy"); !os.IsNotExist(err) {
+               fname := fmt.Sprintf("/tmp/proxy.%d.log", os.Getpid())
+               logFile, _ := os.OpenFile(fname, os.O_WRONLY | os.O_CREATE | os.O_SYNC, 0755)
+               defer logFile.Close()
+               if false {
+                       trace.Start(logFile)
+                       defer trace.Stop()
+               } else {
+                       syscall.Dup2(int(logFile.Fd()), 1)
+                       syscall.Dup2(int(logFile.Fd()), 2)
+               }
+       }
        realMain()
 }

And in the runtime:

diff --git a/virtcontainers/kata_proxy.go b/virtcontainers/kata_proxy.go
index 1ff9d01..6fbc0b7 100644
--- a/virtcontainers/kata_proxy.go
+++ b/virtcontainers/kata_proxy.go
@@ -6,6 +6,7 @@
 package virtcontainers

 import (
+       "os"
        "os/exec"
        "syscall"
 )
@@ -47,6 +48,13 @@ func (p *kataProxy) start(params proxyParams) (int, string, error) {
        }

        cmd := exec.Command(args[0], args[1:]...)
+
+       if _, err := os.Stat("/tmp/trace_proxy"); !os.IsNotExist(err) {
+               env := os.Environ()
+               env = append(env, "GODEBUG=gctrace=2,schedtrace=1000")
+               cmd.Env = env
+       }
+
        if err := cmd.Start(); err != nil {
                return -1, "", err
        }

As per the code, the data then turns up in a file in /tmp. Also, you enable it by creating a file in /tmp. Note, the if false stuff is if you want to enable internal golang trace and have that go into the file - that is a different sort of trace....