A contrived example for trying out "terraform test -junit-xml=..."

.gitignore

/terraform
*.xml

README.md

Example for testing terraform test -junit-xml=...

Terraform CLI v1.8.0-alpha20240131 has introduced an experimental new option for terraform test to output a JUnit XML result report in addition to its other output.

This directory contains a simple Terraform module with some contrived tests which might be useful when testing the new behavior to see how well its output gets rendered in your favorite CI/CD tool, or other test result reporting software.

Preparation

Make sure you're using a Terraform CLI release that has the experiment enabled.

Only alpha releases of Terraform CLI contain experimental features, at the time of writing this the only release to include it is Terraform CLI v1.8.0-alpha20240131. (Other alpha releases with changes to the experiment might follow later.)

Confirm by running terraform version:

$ terraform version
Terraform v1.8.0-alpha20240131
on linux_amd64

(This example was from the linux_amd64 build, but any build of this version, or a later alpha release still containing a compatible version of this experiment, is sufficient.)

Usage

In the directory containing this README file, run the following:

terraform test -junit-xml=junit-result.xml

Assuming that the test run was able to start (i.e. the module and its tests were all statically valid), the command should print its normal test report output to stdout but also write a junit-result.xml file into your current working directory.

The tests are written to pass by default. If you'd like to cause a failure to see how that would be included in the JUnit XML, you can edit the empty_name.tftest.hcl file to remove the expectation for the resource precondition to fail, or make some other change that would cause checks or test assertions to fail.

Please note that the JUnit XML file will be created only if the module and the test scenario files are valid enough to be loaded. As with JUnit XML support in some other languages, Terraform generates the JUnit result report only if the configuration was valid enough for the tests to even begin executing, since otherwise there would not be enough information to even decide which testsuite elements to describe in the report.

Known Quirks in v1.8.0-alpha20240131

The initial experimental release is primarily motivated by finding out if the way we've mapped Terraform's testing model onto the JUnit XML model produces useful and satisfying results in most existing software that consumes this format.

With that in mind, the implementation is quite "bare-bones" and has some known quirks that we would plan to address in a later release if this initial experiment is successful.

Invalid XML Preprocessing Instruction

Unfortunately due to a mistake in preparing the final code for the experiment, the alpha release generates an XML file with an invalid <?xml ... ?> sequence that is missing a trailing quote on the encoding. That is fixed in Terraform CLI's main branch and so will be fixed in the next alpha release, but until then you'll need to manually fix the output before testing.

If you prefer, you could build Terraform CLI from source code on the main branch. If you do that you'll need to build in the mode which compiles in the experiments, like this:

go build -ldflags="-X 'main.experimentsAllowed=yes'" -o terraform

Diagnostics lack source snippets

Terraform uses the word "diagnostics" to refer broadly to all of the error and warning messages it produces in various situations.

The current JUnit XML renderer includes diagnostic information as part of the description of test errors, but currently it doesn't have access to the metadata that Terraform normally uses to insert a source code snippet and some other ancillary context. Therefore you'll find that most error and warning messages currently report that source code is unavailable:

Error: Resource precondition failed

  on example.tf line 14:
  (source code not available)

Name must not be empty.

If we decide to proceed with further development on this feature, we'll make sure that the JUnit XML renderer has access to the needed metadata to match the information that would be returned in the normal human-oriented console output; this current limitation is just a compromise to allow the experiment to be implemented in a more self-contained manner to reduce the risk that it would impact any non-experimental functionality.

Filtering excludes testsuite elements entirely

The terraform test command supports a -filter option for reducing the number of tests being run.

For this initial experiment, the JUnit XML includes only the scenarios that matched the filter and says nothing at all about those that were excluded.

We're unsure whether this is a good decision or whether it would be better to report filtered test scenarios as having been "skipped". We'd appreciate your feedback if you have a situation where including filtered-out test scenarios in the result report would be useful or important.

Not Tested with -cloud-run

To keep the new experimental code as segregated as possible from other code, we've focused for the moment only on the local execution mode of terraform test, and have not tested the behavior when the tests are being run remotely in Terraform Cloud private module registry.

We don't recommend using the -junit-xml=... option in combination with the -cloud-run=... option for the moment. If this experiment is successful, then supporting it for Terraform Cloud remote runs would be a prerequisite for stabilizing it, but we want to make sure that the JUnit XML structure itself is useful first.

empty_name.tftest.hcl

run "empty" {
  command = plan
  variables {
    name = ""
  }

  expect_failures = [
    # Comment out the following line to get an
    # _unexpected_ failure and see how that
    # renders into the JUnit XML.
    terraform_data.example,
  ]
}

example.tf

variable "name" {
  type = string
}

resource "terraform_data" "example" {
  input = {
    name       = var.name
    upper_name = upper(var.name)
  }

  lifecycle {
    precondition {
      condition     = var.name != ""
      error_message = "Name must not be empty."
    }
  }
}

output "upper_name" {
  value = terraform_data.example.output.upper_name

  precondition {
    condition     = !strcontains(terraform_data.example.output.upper_name, "a")
    error_message = "Uppercase name should not contain a lowercase \"a\"."
  }
}

valid.tftest.hcl

run "ok" {
  variables {
    name = "Chiana"
  }
}