krishnadey30/Unit Test Frameworks.md

## Unit Test Frameworks.md

      
    Raw
  

              Unit Test Frameworks.md
            
          
    Unit Test Frameworks

 Julia
 Go
 D

Julia's Unit Testing

Testing Base Julia

Julia provides a Function Base.runtests for testing the Base Julia.
If you build Julia from source, you can run this test suite with make test. In a binary install, you can run the test suite using Base.runtests().
Base.runtests — Function.

Base.runtests(tests=["all"]; ncores=ceil(Int, Sys.CPU_THREADS / 2),
              exit_on_error=false, [seed])
You can list the test in tests , which can be either a string or an array of strings, that will be run using ncoresprocessors. If exit_on_error is false, when one test fails, all remaining tests in other files will still be run; they are otherwise discarded, when exit_on_error == true.
Basic Unit Tests

The Test module provides simple unit testing functionality which can be performed with the @test and @test_throws macros:
Test.@test

@test ex
This macro checks if the expression ex evaluates to true. If it does it returns a Pass result, a Fail result if it is false, and an Error result if it could not be evaluated.
Examples
julia> @test true
Test Passed

julia> @test [1, 2] + [2, 1] == [3, 3]
Test Passed
Test.@test_throws

@test_throws exception expr
This macro checks that the expression expr throws exception. The exception may specify either a type, or a value (which will be tested for equality by comparing fields).
Examples
julia> @test_throws BoundsError [1, 2, 3][4]
Test Passed
      Thrown: BoundsError

julia> @test_throws DimensionMismatch [1, 2, 3] + [1, 2]
Test Passed
      Thrown: DimensionMismatch
Working with Test Sets

The @testset macro can be used to group tests into sets. All the tests in a test set will be run, and at the end of the test set a summary will be printed. If any of the tests failed, or could not be evaluated due to an error, the test set will then throw a TestSetException.
Examples
julia> @testset "trigonometric identities" begin
           θ = 2/3*π
           @test sin(-θ) ≈ -sin(θ)
           @test cos(-θ) ≈ cos(θ)
           @test sin(2θ) ≈ 2*sin(θ)*cos(θ)
           @test cos(2θ) ≈ cos(θ)^2 - sin(θ)^2
       end;
Test Summary:            | Pass  Total
trigonometric identities |    4      4
We can put our tests for the foo(x) function in a test set:
julia> foo(x) = length(x)^2
foo (generic function with 1 method)

julia> @testset "Foo Tests" begin
           @test foo("a")   == 1
           @test foo("ab")  == 4
           @test foo("abc") == 9
       end;
Test Summary: | Pass  Total
Foo Tests     |    3      3
Test sets can also be nested:
julia> @testset "Foo Tests" begin
           @testset "Animals" begin
               @test foo("cat") == 9
               @test foo("dog") == foo("cat")
           end
           @testset "Arrays $i" for i in 1:3
               @test foo(zeros(i)) == i^2
               @test foo(fill(1.0, i)) == i^2
           end
       end;
Test Summary: | Pass  Total
Foo Tests     |    8      8
In the event that a nested test set has no failures, as happened here, it will be hidden in the summary. If we do have a test failure, only the details for the failed test sets will be shown:
julia> @testset "Foo Tests" begin
           @testset "Animals" begin
               @testset "Felines" begin
                   @test foo("cat") == 9
               end
               @testset "Canines" begin
                   @test foo("dog") == 9
               end
           end
           @testset "Arrays" begin
               @test foo(zeros(2)) == 4
               @test foo(fill(1.0, 4)) == 15
           end
       end

Arrays: Test Failed
  Expression: foo(fill(1.0, 4)) == 15
   Evaluated: 16 == 15
[...]
Test Summary: | Pass  Fail  Total
Foo Tests     |    3     1      4
  Animals     |    2            2
  Arrays      |    1     1      2
ERROR: Some tests did not pass: 3 passed, 1 failed, 0 errored, 0 broken.
Other Test Macros

Test.@inferred

@inferred f(x)

This macro checks that the call expression f(x) returns a value of the same type inferred by the compiler. It is useful to check for type stability.
f(x) can be any call expression. It returns the result of f(x) if the types match, and an Error result if it finds different types.
julia> f(a, b, c) = b > 1 ? 1 : 1.0
f (generic function with 1 method)

julia> typeof(f(1, 2, 3))
Int64

julia> @inferred f(1, 2, 3)
ERROR: return type Int64 does not match inferred return type Union{Float64, Int64}
Stacktrace:
[...]

julia> @inferred max(1, 2)
2
Test.@test_logs

@test_logs [log_patterns...] [keywords] expression

Collect a list of log records generated by expression using collect_test_logs, check that they match the sequence log_patterns, and return the value of expression.
Examples
Consider a function which logs a warning, and several debug messages:
function foo(n)
    @info "Doing foo with n=$n"
    for i=1:n
        @debug "Iteration $i"
    end
    42
end
We can test the info message using
@test_logs (:info,"Doing foo with n=2") foo(2)
If we also wanted to test the debug messages, these need to be enabled with the min_level keyword:
@test_logs (:info,"Doing foo with n=2") (:debug,"Iteration 1") (:debug,"Iteration 2") min_level=Debug foo(2)
The macro may be chained with @test to also test the returned value:
@test (@test_logs (:info,"Doing foo with n=2") foo(2)) == 42
Test.@test_deprecated

@test_deprecated [pattern] expression
When --depwarn=yes, test that expression emits a deprecation warning and return the value of expression. The log message string will be matched against pattern which defaults to r"deprecated"i.
When --depwarn=no, simply return the result of executing expression. When --depwarn=error, check that an ErrorException is thrown.
Examples
# Deprecated in julia 0.7
@test_deprecated num2hex(1)

# The returned value can be tested by chaining with @test:
@test (@test_deprecated num2hex(1)) == "0000000000000001"
Test.@test_warn

@test_warn msg expr
Test whether evaluating expr results in stderr output that contains the msg string or matches the msgregular expression. If msg is a Boolean function, tests whether msg(output) returns true. If msg is a tuple or array, checks that the error output contains/matches each item in msg. Returns the result of evaluating expr.
Test.@test_nowarn

@test_nowarn expr
Test whether evaluating expr results in empty stderr output (no warnings or other messages). Returns the result of evaluating expr.
Broken Tests

If a test fails consistently it can be changed to use the @test_broken macro. This will denote the test as Broken if the test continues to fail and alerts the user via an Error if the test succeeds.
Test.@test_broken

@test_broken ex
@test_broken f(args...) key=val ...
Indicates a test that should pass but currently consistently fails. Tests that the expression ex evaluates to false or causes an exception. Returns a Broken Result if it does, or an Error Result if the expression evaluates to true.
The @test_broken f(args...) key=val... form works as for the @test macro.
Examples
julia> @test_broken 1 == 2
Test Broken
  Expression: 1 == 2

julia> @test_broken 1 == 2 atol=0.1
Test Broken
  Expression: ==(1, 2, atol=0.1)
@test_skip is also available to skip a test without evaluation, but counting the skipped test in the test set reporting. The test will not run but gives a Broken Result.
Test.@test_skip

@test_skip ex
@test_skip f(args...) key=val ...
Marks a test that should not be executed but should be included in test summary reporting as Broken. This can be useful for tests that intermittently fail, or tests of not-yet-implemented functionality.
The @test_skip f(args...) key=val... form works as for the @test macro.
Examples
julia> @test_skip 1 == 2
Test Broken
  Skipped: 1 == 2

julia> @test_skip 1 == 2 atol=0.1
Test Broken
  Skipped: ==(1, 2, atol=0.1)
Go's Unit Testing

import "testing"
In Go the package testing provides support for automated testing. It is intended to be used in concert with the go test command, which automates execution of any function of the form
func TestXxx(*testing.T)
where Xxx does not start with a lowercase letter. The function name serves to identify the test routine.
Test Suite

To write a new test suite, create a file whose name ends _test.go that contains the TestXxx functions. The file will be excluded from regular package builds but will be included when the go test command is run.
A simple test function looks like this:
func TestAbs(t *testing.T) {
    got := Abs(-1)
    if got != 1 {
        t.Errorf("Abs(-1) = %d; want 1", got)
    }
}
Benchmarks

Functions of the form
func BenchmarkXxx(*testing.B)
are considered benchmarks, and are executed by the go test command when its -bench flag is provided. Benchmarks are run sequentially.
A sample benchmark function looks like this:
func BenchmarkHello(b *testing.B) {
    for i := 0; i < b.N; i++ {
        fmt.Sprintf("hello")
    }
}
The benchmark function must run the target code b.N times. During benchmark execution, b.N is adjusted until the benchmark function lasts long enough to be timed reliably. The output
BenchmarkHello    10000000    282 ns/op
means that the loop ran 10000000 times at a speed of 282 ns per loop.
Examples

The package also runs and verifies example code. Example functions may include a concluding line comment that begins with Output: and is compared with the standard output of the function when the tests are run. (The comparison ignores leading and trailing space.) These are examples of an example:
func ExampleHello() {
    fmt.Println("hello")
    // Output: hello
}

func ExampleSalutations() {
    fmt.Println("hello, and")
    fmt.Println("goodbye")
    // Output:
    // hello, and
    // goodbye
}
The comment prefix Unordered output: is like Output:, but matches any line order:
func ExamplePerm() {
    for _, value := range Perm(4) {
        fmt.Println(value)
    }
    // Unordered output: 4
    // 2
    // 1
    // 3
    // 0
}
Skipping

Tests or benchmarks may be skipped at run time with a call to the Skip method of *T or *B:
func TestTimeConsuming(t *testing.T) {
    if testing.Short() {
        t.Skip("skipping test in short mode.")
    }
    ...
}
Subtests and Sub-benchmarks

The Run methods of T and B allow defining subtests and sub-benchmarks, without having to define separate functions for each. This enables uses like table-driven benchmarks and creating hierarchical tests. It also provides a way to share common setup and tear-down code:
func TestFoo(t *testing.T) {
    // <setup code>
    t.Run("A=1", func(t *testing.T) { ... })
    t.Run("A=2", func(t *testing.T) { ... })
    t.Run("B=1", func(t *testing.T) { ... })
    // <tear-down code>
}
Each subtest and sub-benchmark has a unique name: the combination of the name of the top-level test and the sequence of names passed to Run, separated by slashes, with an optional trailing sequence number for disambiguation.
The argument to the -run and -bench command-line flags is an unanchored regular expression that matches the test's name.
go test -run ''      # Run all tests.
go test -run Foo     # Run top-level tests matching "Foo", such as "TestFooBar".
go test -run Foo/A=  # For top-level tests matching "Foo", run subtests matching "A=".
go test -run /A=1    # For all top-level tests, run subtests matching "A=1".
Main

It is sometimes necessary for a test program to do extra setup or teardown before or after testing. It is also sometimes necessary for a test to control which code runs on the main thread. To support these and other cases, if a test file contains a function:
func TestMain(m *testing.M)
then the generated test will call TestMain(m) instead of running the tests directly. TestMain runs in the main goroutine and can do whatever setup and teardown is necessary around a call to m.Run. It should then call os.Exit with the result of m.Run.
A simple implementation of TestMain is:
func TestMain(m *testing.M) {
	// call flag.Parse() here if TestMain uses flags
	os.Exit(m.Run())
}
D's Unit Testing

Unit tests are a special function defined like:
unittest
{
    ...test code...
}
Individual tests are specified in the unit test using AssertExpressions. Unlike AssertExpressions used elsewhere, the assert is not assumed to hold, and upon assert failure the program is still in a defined state.
There can be any number of unit test functions in a module, including within struct, union and class declarations. They are executed in lexical order.
Unit tests, when enabled, are run after all static initialization is complete and before the main() function is called.
For example, given a class Sum that is used to add two values, a unit test can be given:
class Sum
{
    int add(int x, int y) { return x + y; }

    unittest
    {
        Sum sum = new Sum;
        assert(sum.add(3,4) == 7);
        assert(sum.add(-2,0) == -2);
    }
}
Attributed Unittests

A unittest may be attributed with any of the global function attributes. Such unittests are useful in verifying the given attribute(s) on a template function:
void myFunc(T)(T[] data)
{
    if (data.length > 2)
        data[0] = data[1];
}

@safe nothrow unittest
{
    auto arr = [1,2,3];
    myFunc(arr);
    assert(arr == [2,2,3]);
}
This unittest verifies that myFunc contains only @safe, nothrow code.
Implementation Defined


If unit tests are not enabled, the implementation is not required to check the UnitTest for syntactic or semantic correctness. This is to reduce the compile time impact of larger unit test sections. The tokens must still be valid, and the implementation can merely count { and } tokens to find the end of the UnitTest's BlockStatement.
Use of a compiler switch such as -unittest to enable them is suggested.

Documented Unittests

Documented unittests allow the developer to deliver code examples to the user, while at the same time automatically verifying that the examples are valid. This avoids the frequent problem of having outdated documentation for some piece of code.
If a declaration is followed by a documented unittest, the code in the unittest will be inserted in the example section of the declaration:
/// Math class
class Math
{
    /// add function
    static int add(int x, int y) { return x + y; }

    ///
    unittest
    {
        assert(add(2, 2) == 4);
    }
}

///
unittest
{
    auto math = new Math();
    auto result = math.add(2, 2);
}
The above will generate the following documentation:
class Math;
	Math class
	Example:
auto math = new Math; 
auto result = math.add(2, 2); 
	int add(int x, int y);
		add function
		Example:
assert(add(2, 2) == 4); 
A unittest which is not documented, or is marked as private will not be used to generate code samples.
There can be multiple documented unittests and they can appear in any order. They will be attached to the last non-unittest declaration:
/// add function
int add(int x, int y) { return x + y; }

/// code sample generated
unittest
{
    assert(add(1, 1) == 2);
}

/// code sample not generated because the unittest is private
private unittest
{
    assert(add(2, 2) == 4);
}

unittest
{
    /// code sample not generated because the unittest isn't documented
    assert(add(3, 3) == 6);
}

/// code sample generated, even if it only includes comments (or is empty)
unittest
{
    /** assert(add(4, 4) == 8); */
}
The above will generate the following documentation:
int add(int x, int y);
add function
	Examples:
	code sample generated
assert(add(1, 1) == 2);
	Examples:
	code sample generated, even if it is empty or only includes comments
/** assert(add(4, 4) == 8); */