gofedlib vs go list

go-ingestion-docs.md

Document describing what kind of dependencies go list reports, and whether or not it satisfies the requirements of ingestion.

An alternative approach ( as suggested on mailing list) could be to use go list command line tool to find all the dependencies of a Go project.

For example:

$ go get github.com/mattermost/platform
$ go list -f="{{.ImportPath}} {{.Imports}}" github.com/mattermost/platform | python2 -c 'import sys; import re; import json; print json.dumps({x[0]: x[1] for x in map(lambda s: (re.match("(\S+)\s\[(.*?)\]", s).groups()[0], re.match("(\S+)\s\[(.*?)\]", s).groups()[1].split()), sys.stdin.readlines())})'  | python -mjson.tool
{
    "github.com/mattermost/platform": [
        "flag",
        "fmt",
        "github.com/alecthomas/log4go",
        "github.com/go-ldap/ldap",
        "github.com/mattermost/platform/api",
        "github.com/mattermost/platform/einterfaces",
        "github.com/mattermost/platform/manualtesting",
        "github.com/mattermost/platform/model",
        "github.com/mattermost/platform/model/gitlab",
        "github.com/mattermost/platform/utils",
        "github.com/mattermost/platform/web",
        "io/ioutil",
        "net/http",
        "net/url",
        "os",
        "os/signal",
        "runtime",
        "strconv",
        "strings",
        "syscall",
        "time"
    ]
}

Part 1

Here is a comparison of go list vs gofedlib.

Lets us try to find dependencies of github.com/docker/docker using go list:

$ go get github.com/docker/docker
$ go list -f="{{.ImportPath}} {{.Imports}}" github.com/docker/docker
can't load package: package github.com/docker/docker: no Go files in /home/saleem/go/src/github.com/docker/docker 

So it seems that go list tries to check *.go files in base input path. If there are no such *.go files, then no dependencies are reported. Instead an error is generated.

Now lets try gofedlib-cli:

$ go get github.com/docker/docker
$ ./gofedlib-cli -j -d -t -m $GOPATH/src/github.com/docker/docker 2>/dev/null
{
  "deps-main": [
    "https://github.com/docker/go-units",
    "https://github.com/spf13/cobra",
    "https://github.com/coreos/go-systemd",
    "https://github.com/bfirsh/funker-go",
    "https://github.com/golang/sys",
    "https://github.com/moby/buildkit",
    "https://github.com/pkg/errors",
    "https://github.com/docker/go-metrics",
    "https://github.com/docker/distribution",
    "https://github.com/docker/go-connections",
    "https://github.com/docker/docker",
    "https://github.com/docker/swarmkit",
    "https://github.com/spf13/pflag",
    "https://github.com/docker/libnetwork",
    "https://github.com/sirupsen/logrus"
  ],
  "deps-packages": [
    "https://github.com/spf13/cobra",
    "https://github.com/docker/swarmkit",
    "https://github.com/mistifyio/go-zfs",
    "https://github.com/Azure/go-ansiterm",
    "https://github.com/bsphere/le_go",
    "https://github.com/containerd/containerd",
    "https://github.com/opencontainers/runc",
    "https://github.com/moby/buildkit",
    "https://github.com/vdemeester/shakers",
    "https://github.com/vbatts/tar-split",
    "https://github.com/docker/go-metrics",
    "https://github.com/seccomp/libseccomp-golang",
    "https://github.com/gorilla/mux",
    "https://github.com/Graylog2/go-gelf",
    "https://github.com/hashicorp/go-immutable-radix",
    "https://github.com/vishvananda/netlink",
    "https://golang.org/x/sync",
    "https://github.com/gogo/protobuf",
    "https://github.com/docker/go-connections",
    "https://github.com/Nvveen/Gotty",
    "https://github.com/opencontainers/go-digest",
    "https://github.com/coreos/go-systemd",
    "https://github.com/aws/aws-sdk-go",
    "https://github.com/golang/net",
    "https://github.com/golang/sys",
    "https://github.com/docker/distribution",
    "https://github.com/docker/libkv",
    "https://github.com/fluent/fluent-logger-golang",
    "https://github.com/Microsoft/hcsshim",
    "https://github.com/docker/docker",
    "https://github.com/docker/go-units",
    "https://github.com/sirupsen/logrus",
    "https://github.com/syndtr/gocapability",
    "https://github.com/containerd/continuity",
    "https://github.com/fsnotify/fsnotify",
    "https://github.com/docker/libnetwork",
    "https://github.com/grpc/grpc-go",
    "https://github.com/tchap/go-patricia",
    "https://github.com/opencontainers/runtime-spec",
    "https://github.com/stretchr/testify",
    "https://github.com/tonistiigi/fifo",
    "https://github.com/docker/libtrust",
    "https://github.com/Microsoft/go-winio",
    "https://github.com/boltdb/bolt",
    "https://github.com/opencontainers/image-spec",
    "https://github.com/tonistiigi/fsutil",
    "https://github.com/opencontainers/selinux",
    "https://github.com/gotestyourself/gotestyourself",
    "https://github.com/go-check/check",
    "https://github.com/golang/protobuf",
    "https://github.com/Microsoft/opengcs",
    "https://github.com/hashicorp/go-memdb",
    "https://github.com/RackSec/srslog",
    "https://github.com/imdario/mergo",
    "https://github.com/google/go-genproto",
    "https://github.com/pkg/errors",
    "https://github.com/prometheus/client_golang",
    "https://github.com/mattn/go-shellwords",
    "https://github.com/golang/time",
    "https://github.com/spf13/pflag"
  ],
  "deps-tests": [
    "https://github.com/docker/go-units",
    "https://github.com/docker/swarmkit",
    "https://github.com/opencontainers/runc",
    "https://github.com/golang/net",
    "https://github.com/vbatts/tar-split",
    "https://github.com/gotestyourself/gotestyourself",
    "https://github.com/gorilla/mux",
    "https://github.com/vishvananda/netlink",
    "https://github.com/pborman/uuid",
    "https://golang.org/x/sync",
    "https://github.com/gogo/protobuf",
    "https://github.com/docker/go-connections",
    "https://github.com/opencontainers/go-digest",
    "https://github.com/aws/aws-sdk-go",
    "https://github.com/golang/sys",
    "https://github.com/cloudflare/cfssl",
    "https://github.com/docker/distribution",
    "https://github.com/docker/libkv",
    "https://github.com/docker/docker",
    "https://github.com/sirupsen/logrus",
    "https://github.com/containerd/continuity",
    "https://github.com/fsnotify/fsnotify",
    "https://github.com/docker/libnetwork",
    "https://github.com/stretchr/testify",
    "https://github.com/docker/libtrust",
    "https://github.com/Microsoft/go-winio",
    "https://github.com/opencontainers/image-spec",
    "https://github.com/moby/buildkit",
    "https://github.com/kr/pty",
    "https://github.com/go-check/check",
    "https://github.com/opencontainers/selinux",
    "https://github.com/RackSec/srslog",
    "https://github.com/pkg/errors",
    "https://github.com/spf13/pflag"
  ]
}

Clearly go list can not provide dependencies work for multi-module projects by default.

However if we know in advance which base folders contain Go sources, then we can provide those base folders ( or package paths relative to $GOPATH/src ), and use go list as follows:

$ go list -f="{{.ImportPath}} {{.Imports}}" github.com/docker/docker/registry github.com/docker/docker/client | python2 -c 'import sys; import re; import json; print json.dumps({x[0]: x[1] for x in map(lambda s: (re.match("(\S+)\s\[(.*?)\]", s).groups()[0], re.match("(\S+)\s\[(.*?)\]", s).groups()[1].split()), sys.stdin.readlines())})'  | python -mjson.tool
{
    "github.com/docker/docker/client": [
        "bufio",
        "bytes",
        "crypto/tls",
        "encoding/base64",
        "encoding/json",
        "errors",
        "fmt",
        "github.com/docker/docker/vendor/github.com/docker/distribution/reference",
        "github.com/docker/docker/api",
        "github.com/docker/docker/api/types",
        "github.com/docker/docker/api/types/container",
        "github.com/docker/docker/api/types/events",
        "github.com/docker/docker/api/types/filters",
        "github.com/docker/docker/api/types/image",
        "github.com/docker/docker/api/types/network",
        "github.com/docker/docker/api/types/registry",
        "github.com/docker/docker/api/types/swarm",
        "github.com/docker/docker/api/types/time",
        "github.com/docker/docker/api/types/versions",
        "github.com/docker/docker/api/types/volume",
        "github.com/docker/docker/vendor/github.com/docker/go-connections/sockets",
        "github.com/docker/docker/vendor/github.com/docker/go-connections/tlsconfig",
        "github.com/docker/docker/vendor/github.com/opencontainers/go-digest",
        "github.com/docker/docker/vendor/github.com/pkg/errors",
        "github.com/docker/docker/vendor/golang.org/x/net/context",
        "github.com/docker/docker/vendor/golang.org/x/net/context/ctxhttp",
        "io",
        "io/ioutil",
        "net",
        "net/http",
        "net/http/httputil",
        "net/url",
        "os",
        "path",
        "path/filepath",
        "regexp",
        "strconv",
        "strings",
        "time"
    ],
    "github.com/docker/docker/registry": [
        "bytes",
        "crypto/sha256",
        "crypto/sha512",
        "crypto/tls",
        "encoding/hex",
        "encoding/json",
        "errors",
        "fmt",
        "github.com/docker/docker/vendor/github.com/docker/distribution/reference",
        "github.com/docker/docker/vendor/github.com/docker/distribution/registry/api/errcode",
        "github.com/docker/docker/vendor/github.com/docker/distribution/registry/client/auth",
        "github.com/docker/docker/vendor/github.com/docker/distribution/registry/client/auth/challenge",
        "github.com/docker/docker/vendor/github.com/docker/distribution/registry/client/transport",
        "github.com/docker/docker/api/types",
        "github.com/docker/docker/api/types/registry",
        "github.com/docker/docker/pkg/ioutils",
        "github.com/docker/docker/pkg/jsonmessage",
        "github.com/docker/docker/pkg/stringid",
        "github.com/docker/docker/pkg/tarsum",
        "github.com/docker/docker/registry/resumable",
        "github.com/docker/docker/vendor/github.com/docker/go-connections/sockets",
        "github.com/docker/docker/vendor/github.com/docker/go-connections/tlsconfig",
        "github.com/docker/docker/vendor/github.com/pkg/errors",
        "github.com/docker/docker/vendor/github.com/sirupsen/logrus",
        "github.com/docker/docker/vendor/golang.org/x/net/context",
        "io",
        "io/ioutil",
        "net",
        "net/http",
        "net/http/cookiejar",
        "net/url",
        "os",
        "path/filepath",
        "regexp",
        "strconv",
        "strings",
        "sync",
        "time"
    ]
}

For now gofedlib-cli is seems good enough ( provide the issues mentioned are fixed and approved ).

Also using native go list will be the best option as it will be fast and supported native.

Part 2

There is one more significant difference between go list output and gofedlib-cli output, that former just reports the package names. And the later one reports the SCM repo URL, which is better for fetching dependencies.

Using go list a kind of mapping from package name to SCM repo URL will be needed.

This kind of mapping is also used by gofedlib-cli, used to generate the SCM repo URLS - https://github.com/gofed/gofedlib/blob/master/gofedlib/providers/data/ip2pp_mapping.json

Here is a top list of Go packages. https://gist.github.com/tuxdna/37c2b4d36f877e7db387c2fbbe668a65

We can only import with certainty only github.com prefixed packages, and of those only those which are of github.com/<SOME_STRING1>/<SOME_STRING2> form

For packages like github.com/aws/aws-sdk-go/aws we have to handle it properly, i.e tell to the workers to fetch base repo github.com/aws/aws-sdk-go/ and then only work on aws/ sub-folder to process the given package.

Essentially, we fetch the SCM ( or git ) repo for the corresponding package first, and the move the the sub-folder where the target Go package code is located.