Author(s): Nodir Turakulov <nodir@google.com>
With initial input by Russ Cox, Caleb Spare, Andrew Gerrand and Minux Ma.
Last updated: 2015-10-07
Discussion at https://golang.org/issue/2981.
Add -json
flag to go test
.
When specified, go test
stdout is JSON format.
There is a clear need in parsing test and benchmark results by third party
tools, see feedback in https://golang.org/issue/2981.
Currently go test
output format is suited for humans, but not computers.
Also a change to the current format may break existing programs that parse
go test
output.
Currently, under certain conditions, go test
streams test/benchmark results
so a user can see them as they happen.
This proposal attempts to preserve streaming capability in the -json
mode, so
third party tools interpreting go test
output can stream results too.
-json
flag was originally proposed by Russ Cox in
https://golang.org/issue/2981 in 2012.
This proposal differs from the original:
- supports streaming
go test
JSON output contains unrecognized test binary output.- no changes to
testing.InternalTest
andtesting.InternalBenchmark
.
I propose the following user-visible changes:
go test
: add-json
flag.-json
: allgo test
stdout is indented JSON objects containing test binary artifacts, separated by newline. Format below.-json -v
: verbose messages are printed to stderr, so stdout contains only JSON.-json -n
: not supported-json -x
: not supported
testing
package- Add
type TestResult
andtype TestState
. - Add
func Test(f func(*T)) TestResult
to be consistent withfunc Benchmark(f func(*B)) BenchmarkResult
. This is not required for JSON output. - Add
Name
,Output
andProcs
fields toBenchmarkResult
.
- Add
Type definitions and details below.
// TestState is one of terminal test states.
// Implements fmt.Stringer, json.Marshaler and json.Unmarshaler.
type TestState int
const (
PASS TestState = iota
FAIL
SKIP
)
// The results of a test run.
type TestResult struct {
Name string
State TestState
T time.Duration // The total time taken.
Output string // The log created by calling (*T).Log and (*T).Logf.
}
// Test runs a test function and returns results.
func Test(f func(*T)) TestResult
type BenchmarkResult struct {
Name string
Procs int // The value of runtime.GOMAXPROCS for this benchmark run.
Output string // The log created by calling (*B).Log and (*B).Logf.
// existing fields
}
// result is used for test binary output format.
// It is not added to the public API.
//
// Each time a test/benchmark completes, the test binary emits one result
// in unindented JSON format to stdout, surrounded by '\n'.
type result struct {
Test *TestResult `json:",omitempty"`
Benchmark *BenchmarkResult `json:",omitempty"`
}
Example of a test binary stdout (JSON output is made indented for the convenience of the reader. It will be unindented in fact):
{
"Test": {
"Name": "TestFoo",
"State": "PASS",
"T": 1000000
}
}
Random string written directly to os.Stdout.
{
"Tests": {
"Name": "TestBar",
"State": "PASS",
"T": 1000000,
"Output": "some test output\n"
}
}
{
"Benchmark": {
"Name": "BenchmarkBar",
"State": "PASS",
"T": 1000000,
"N": 1000,
"Bytes": 0,
"MemAllocs": 0,
"MemBytes": 0
}
}
go test
JSON output format:
// TestResult contains one output line of a test binary.
type TestResult struct {
Package string // package of the test binary.
Test *TestResult `json:",omitempty"`
Benchmark *BenchmarkResult `json:",omitempty"`
Stdout string `json:",omitempty"` // Unrecognized stdout of the test binary.
Stderr string `json:",omitempty"` // Stderr output line of the test binary.
}
Example go test -json
output
{
"Package": "example.com/foobar",
"Test": {
"Name": "TestFoo",
"State": "PASS",
"T": 1000000
}
}
{
"Package": "example.com/foobar",
"Stdout": "Random string written directly to os.Stdout.\n"
}
{
"Package": "example.com/foobar",
"Test": {
"Name": "TestBar",
"State": "PASS",
"T": 1000000,
"Output": "some test output\n"
}
}
{
"Package": "example.com/foobar",
"Benchmark": {
"Name": "BenchmarkBar",
"Procs": 8,
"T": 1000000,
"N": 1000,
"Bytes": 0,
"MemAllocs": 0,
"MemBytes": 0
}
}
- A test binary surrounds
testing.result
JSON with\n
to handle situation when a string without a trailing\n
is printed directly toos.Stdout
. - A test binary always streams results so we don't loose them if the binary panics.
Alternatives:
- Add
-format
and-benchformat
flags proposed in golang/go#12826. This is simpler to implement by moving the burden of output parsing to third party programs.
Trade offs:
-
I propose to make
-json
mutually exclusive with-n
and-x
flags. This is a trade off forgo test
output format simplicity. Supporting-json
with-n
/-x
flags would require a new field inTestResult
that would contain commands that have been run. Note that we cannot print commands to stdout because stdout must be valid JSON.Supporting
-json
with-n
/-x
flags would also raise the question whether the field must be specific to commands or it should contain anythingbuild.go
prints to stdout. At this time-n
and-x
are the only flags that causebuild.go
to print to stdout, so we can avoid the problem for now.If we add more output to
build.go
in future, we can addBuildOutput string
field toTestResult
incmd/go/test.go
for arbitrarybuild.go
output.I propose not to add
BuildOutput
now because-n
affectsgo test
too. For example,go test -n
prints a command to run the test binary, which should not be a part ofBuildOutput
(because it is not build). -
With
Stdout
andStderr
fields separated ingo test
output format, it is impossible to determine the order of the test binary output. This is a trade off forgo test
output format simplicity.Combining
Stdout
andStderr
would make it impossible to distinguish stdout and stderr.If a third party tool needs to know the stdout/stderr order, it can leverage the
-exec
flag.
The API changes are fully backwards compatible.
Most of the work would be done by the author of this proposal.
Implementation steps:
-
Add new fields to
testing.BenchmarkResult
. Modifytesting.(*B).launch
to fill the new fields. -
Add
type TestResult
,type TestStatus
andfunc Test(f func(*T)) TestResult
to packagetesting
. Modifytesting.tRunner
to createTestResult
. -
Add
-test.json
flag to thetesting
package. Modifytesting.(*T).report
andtesting.RunBenchmarks
functions to print JSON if-test.json
is specified. If-test.verbose
was passed, print verbose messages to stderr. -
Add
-json
flag togo test
. If-json
is passed, pass-test.json
to test binaries.For each line in a test binary output, try to parse it as
testing.Result
in JSON format. Accumulate onetesting.Result
per package.If
-stream
was specified, overridetestStreamOutput
variable value. Print JSON output on each test binary output line. If not streaming, print one JSON with all artifacts on completion of all test binaries.
The goal is to get agreement on this proposal and to complete the work before the 1.6 freeze date.