Skip to content

Instantly share code, notes, and snippets.

@nodirt

nodirt/proposal.md

Last active September 14, 2016 15:54
Show Gist options
  • Save nodirt/ecf2ca7ad63366e8859b3294669d22b2 to your computer and use it in GitHub Desktop.
Save nodirt/ecf2ca7ad63366e8859b3294669d22b2 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
proposal.md

Proposal: -json flag in go test

Author(s): Nodir Turakulov <nodir@google.com>

With initial input by Russ Cox, Caleb Spare, Andrew Gerrand and Minux Ma.

Last updated: 2016-08-26

Discussion at https://golang.org/issue/2981.

Abstract

Add -json flag to go test. When specified, go test stdout is JSON.

Background

There is a clear need in parsing test and benchmark results by third party tools, see feedback in https://golang.org/issue/2981. Currently go test output format is suited for humans, but not computers. Also a change to the current format may break existing programs that parse go test output.

Currently, under certain conditions, go test streams test/benchmark results so a user can see them as they happen. Also streaming prevents loosing data if go test crashes. This proposal attempts to preserve streaming capability in the -json mode, so third party tools interpreting go test output can stream results too.

-json flag was originally proposed by Russ Cox in https://golang.org/issue/2981 in 2012. This proposal has several differences.

Proposal

I propose the following user-visible changes:

  • add -json flag to go test
    • -json: go test stdout is a valid JSON Text Sequence of JSON objects containing test binary artifacts. Format below.
    • -json -v: verbose messages are printed to stderr, so stdout contains only JSON.
    • -json -n: not supported
    • -json -x: not supported
  • In testing package
    • Add type State which is an enum
    • Add type JSONResult for JSON output.
    • Change Cover.CoveredPackages field type from string to []string.

Type definitions and details below.

testing package

// State is one of test/benchmark execution states.
// Implements fmt.Stringer, json.Marshaler and json.Unmarshaler.
type State int

const (
    // RUN means a test/benchmark execution has started
    RUN State = iota + 1
    PASS
    FAIL
    SKIP
)

// JSONResult structs encoded in JSON are emitted by `go test` if -json flag is
// specified.
type JSONResult struct {
    // Configuration is metadata produced by test/benchmark infrastructure.
    // The interpretation of a key/value pair is up to tooling, but the key/value
    // pair is considered to describe all test/benchmark results that follow,
    // until overwritten by a JSONResult with a non-empty Configuration field.
    //
    // The key begins with a lowercase character (as defined by unicode.IsLower),
    // contains no space characters (as defined by unicode.IsSpace)
    // nor upper case characters (as defined by unicode.IsUpper).
    // Conventionally, multiword keys are written with the words separated by hyphens,
    // as in cpu-speed.
    Configuration map[string]string

    // Package is a full name of the package containing the test/benchmark.
    // It is zero iff Name is zero.
    Package string  `json:",omitempty"
    // Name is the name of the test/benchmark that this JSONResult is about.
    Name    string  `json:",omitempty"
    // State is the current state of the test/benchmark.
    // It is non-zero iff Name is non-zero.
    State   State   `json:",omitempty"
    // Procs is the value of runtime.GOMAXPROCS for this test/benchmark run.
    // It is specified only in the first JSONResult of a test/benchmark.
    Procs   int     `json:",omitempty"
    // Log is log created by calling Log or Logf functions of *T or *B.
    // A JSONResult with Log is emitted by go test as soon as possible.
    // First occurrence of test/benchmark does not contain logs.
    Log     string  `json:",omitempty"

    // Benchmark contains benchmark-specific details.
    // It is emitted in the final JSONResult of a benchmark with a terminal
    // State if the benchmark does not have sub-benchmarks.
    Benchmark *BenchmarkResult  `json:",omitempty"

    CoverageMode     string    `json:",omitempty"
    TotalStatements  int64     `json:",omitempty"
    ActiveStatements int64     `json:",omitempty"
    CoveredPackages  []string  `json:",omitempty"

    // Stdout is text written by the test binary directly to os.Stdout.
    // If this field is non-zero, others are zero.
    Stdout string  `json:",omitempty"
    // Stderr is text written by test binary directly to os.Stderr.
    // If this field is non-zero, others are zero.
    Stderr string  `json:",omitempty"
}

Example output

Here is an example of go test -json output. It is simplified and commented for the convenience of the reader; in practice it will be unindented, will contain JSON Text Sequence separators and no comments.

// go test emits environment configuration
{
    "Configuration": {
        "commit": "7cd9055",
        "commit-time": "2016-02-11T13:25:45-0500",
        "goos": "darwin",
        "goarch": "amd64",
        "cpu": "Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz",
        "cpu-count": "8",
        "cpu-physical-count": "4",
        "os": "Mac OS X 10.11.3",
        "mem": "16 GB"
    }
}
// TestFoo started
{
    "Package": "github.com/user/repo",
    "Name": "TestFoo",
    "State": "RUN",
    "Procs": 4
}
// A line was written directly to os.Stdout
{
    "Package": "github.com/user/repo",
    "Stderr": "Random string written directly to os.Stdout\n"
}
// TestFoo passed
{
    "Package": "github.com/user/repo",
    "Name": "TestFoo",
    "State": "PASS",
}
// TestBar started
{
    "Package": "github.com/user/repo",
    "Name": "TestBar",
    "State": "RUN",
    "Procs": 4
}
// TestBar logged a line
{
    "Package": "github.com/user/repo",
    "Name": "TestBar",
    "State": "RUN",
    "Log": "some test output\n"
}
// TestBar failed
{
    "Package": "github.com/user/repo",
    "Name": "TestBar",
    "State": "FAIL"
}
// TestComposite started
{
    "Package": "github.com/user/repo",
    "Name": "TestComposite",
    "State": "RUN",
    "Procs": 4
}
// TestComposite/A=1 subtest started
{
    "Package": "github.com/user/repo",
    "Name": "TestComposite/A=1",
    "State": "RUN",
    "Procs": 4
}
// TestComposite/A=1 passed
{
    "Package": "github.com/user/repo",
    "Name": "TestComposite/A=1",
    "State": "PASS",
}
// TestComposite passed
{
    "Package": "github.com/user/repo",
    "Name": "TestComposite",
    "State": "PASS",
}
// Example1 started
{
    "Package": "github.com/user/repo",
    "Name": "Example1",
    "State": "RUN",
    "Procs": 4
}
// Example1 passed
{
    "Package": "github.com/user/repo",
    "Name": "Example1",
    "State": "PASS"
}
// BenchmarkRun started
{
    "Package": "github.com/user/repo",
    "Name": "BenchmarkBar",
    "State": "RUN",
    "Procs": 4
}
// BenchmarkRun passed
{
    "Package": "github.com/user/repo",
    "Name": "BenchmarkBar",
    "State": "PASS",
    "Benchmark": {
        "T": 1000000,
        "N": 1000,
        "Bytes": 100,
        "MemAllocs": 10,
        "MemBytes": 10
    }
}
// BenchmarkComposite started
{
    "Package": "github.com/user/repo",
    "Name": "BenchmarkComposite",
    "State": "RUN",
    "Procs": 4
}
// BenchmarkComposite/A=1 started
{
    "Package": "github.com/user/repo",
    "Name": "BenchmarkComposite/A=1",
    "State": "RUN",
    "Procs": 4
}
// BenchmarkComposite/A=1 passed
{
    "Package": "github.com/user/repo",
    "Name": "BenchmarkComposite/A=1",
    "State": "PASS",
    "Benchmark": {
        "T": 1000000,
        "N": 1000,
        "Bytes": 100,
        "MemAllocs": 10,
        "MemBytes": 10
    }
}
// BenchmarComposite passed
{
    "Package": "github.com/user/repo",
    "Name": "BenchmarComposite",
    "State": "PASS"
}
// Total coverage information in the end.
{
    "CoverageMode": "set",
    "TotalStatements":  1000,
    "ActiveStatements": 900,
    "CoveredPackages": [
        "github.com/user/repo"
    ]
}

Rationale

Alternatives:

  • Add -format and -benchformat flags proposed in golang/go#12826. This is simpler to implement by moving the burden of output parsing to third party programs.

Trade offs:

  • I propose to make -json mutually exclusive with -n and -x flags. These flags belong to go build subcommand while this proposal is scoped to go test. Supporting the flags would require adding JSON output knowledge to go/build.go.

  • JSONResult.Benchmark.T provides duration of a benchmark run, but there is no an equivalent for a test run. This is a trade off for JSONResult simplicity. We don't have to define TestResult because JSONResult is enough to describe a test result.

    Currently go test does not provide test timing info, so the proposal is consistent with the current go test output.

Compatibility

The only backwards incompatibility is changing testing.Cover.CoveredPackages field type, but testing.Cover is not covered by Go 1 compatibility guidelines.

Implementation

Most of the work would be done by the author of this proposal.

Implementation details:

  • To capture text written to stdout/stderr, replace os.Stdout and os.Stderr with os.Pipes.

The goal is to get agreement on this proposal and to complete the work before the 1.8 freeze date.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment