An example project, named 'llvm-project/examples', that demonstrates a small LLVM subproject that runs lit tests that include googletest executables. (Pretend "-" in file names are "/".)

CMakeLists.txt

# The CMake files in this project contain several additional checks, or examples
# of how to use LLVM CMake functions. These additional checks can be disabled by
# setting this option to 'OFF'.
option(EXAMPLES_CMAKE_TESTS
       "Enable tests related to the example project's CMake configuration." ON)

# Most LLVM projects define '<PROJECTNAME>_SOURCE_DIR' and '_BINARY_DIR'
# in their CMake files. These can be used in many ways, such as to
# specify the path to our tests in our lit test config, lit.site.cfg.py.in.
set(EXAMPLES_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR})
set(EXAMPLES_BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR})

# LLVM defines a few top-level CMake options to control how test targets are
# generated or built, in 'llvm/CMakeLists.txt':
#
#   * 'LLVM_BUILD_TESTS' controls whether test targets are built as part of the
#     default CMake build target -- again, that's an invocation of just 'make',
#     without any target specified explicitly. This is 'OFF' by default.
#   * 'LLVM_INCLUDE_TESTS' controls whether CMake generates test targets in the
#     first place. When this is 'OFF', you couldn't ask 'make' or 'ninja' to
#     build a test target even by its specific name, because that target won't
#     exist. This is 'ON' by default, so test targets are generated. The entire
#     'llvm/test' and 'llvm/utils/lit' directories are gated by this option, so
#     a user could turn it 'OFF' to save some time on CMake configuration.
#     Similarly, we gate the examples test directories here.
if(LLVM_INCLUDE_TESTS)
  add_subdirectory(test)
  add_subdirectory(unittests)
endif()

test-CMakeLists.txt

# This directory contains the project's test suite. The tests are run using an
# LLVM tool, written in Python, called 'lit'. 'lit' is run by passing it the
# path to a directory. It will search that directory for a file conventionally
# named 'lit.cfg.py', which instructs lit on:
#
#   1. Which directory to recursively search for test files
#   2. What file extensions test files in those directories will have
#
# ...and so on.
#
# lit can be configured to use one of two built-in "test runners" (or a
# completely custom one if you write a class in Python), to run either
# googletest unit tests, or tests that define a series of commands to run.
# For these "shell test" files, it will find each source line that begins with
# 'RUN:', and it will execute the statement that follows it as if it were a
# shell command, asserting that the command exits with a 0 exit code.
#
# So, for example, the following is a valid 'lit' test that will pass when run:
#
#     # RUN: echo "hello" > tmp.txt
#     # RUN: cat tmp.txt | grep "he"
#
# lit also substitutes special markers, typically prefaced with a '%', with
# values such as the path to the current file. For example, lit substitutes
# '%t' with the path to a unique temporary file. Using that, we can re-write our
# test above like so:
#
#     # RUN: echo "hello" > %t.txt
#     # RUN: cat %t.txt | grep "he"
#
# Typically, tests in LLVM projects don't use the system's 'grep' command to
# match text, instead using the LLVM utility 'FileCheck'. FileCheck tests an
# input stream against expectations written in a file that are prefaced with a
# 'CHECK:' directive. For example, the test above can be re-written as below
# to achieve identical semantics (note that lit substitutes '%s' with the path
# to the current file):
#
#     # RUN: echo "hello" | FileCheck %s
#     # CHECK: he
#
# In this CMake file, we use CMake functions defined by LLVM to set up two lit
# test suites:
#
#   1. A suite for "shell tests", which are stored in this 'test' source
#      directory, and whose 'lit.cfg.py' config file will be copied to the
#      build folder's 'test' directory.
#   2. A suite for unit tests defined with googletest, which are stored in the
#      'examples/unittests' directory. The 'lit.cfg.py' config file for these
#      is defined in the 'examples/test/Unit' directory, and will be copied to
#      the build folder's 'test/Unit' directory.

# 'configure_lit_site_cfg' is defined in llvm/cmake/modules/AddLLVM.cmake.
# It copies an input file from one location (the first positional argument) to
# another (the second positional argument), modifying its contents.
#
# It does so using CMake's native 'configure_file' function, which copies a
# file and replaces any variables '@LIKE_THIS@' with that variable's value in
# CMake. So, for example, here we copy lit.site.cfg.py.in from this source
# directory to our build's binary directory, replacing the
# '@EXAMPLES_BINARY_DIR@' in that file with the value we set in
# 'examples/CMakeLists.txt'.
#
# 'configure_lit_site_cfg' mostly just calls CMake's 'configure_file', but
# before it does, it also defines a bunch of CMake variables that our config
# file might need, like 'LLVM_SOURCE_DIR' and 'HOST_OS'.
#
# (It also does some stuff to support 'gn', a tool that some people use to
# build LLVM. I don't know much about that. The named argument, 'MAIN_CONFIG',
# is used by that tool, and apparently should refer to our "main" lit config
# file. More on lit configs below.)
#
# We begin here by copying the configuration file for our first test suite, for
# our "shell tests":
configure_lit_site_cfg(
  # Copy our config file from our current source directory 'examples/test'...
  ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.py.in
  # ...to our output binary directory, 'build/tools/examples/test'.
  ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg.py
  # This parameter is used by the 'gn' build system, apparently.
  MAIN_CONFIG ${CMAKE_CURRENT_SOURCE_DIR}/lit.cfg.py)

# Next we copy the configuration file for our second test suite, for googletest
# unit tests:
configure_lit_site_cfg(
  ${CMAKE_CURRENT_SOURCE_DIR}/Unit/lit.site.cfg.py.in
  ${CMAKE_CURRENT_BINARY_DIR}/Unit/lit.site.cfg.py
  MAIN_CONFIG ${CMAKE_CURRENT_SOURCE_DIR}/Unit/lit.cfg.py)

# Most LLVM projects define a CMake target named 'check-<projectname>'. For
# example, there's 'check-llvm' or 'check-clang'. Below, we use the LLVM CMake
# function 'add_lit_testsuite' (defined in 'llvm/cmake/modules/AddLLVM.cmake')
# to define a 'check-examples' target.
#
# First, we want a user who builds the 'check-examples' target to automatically
# build the dependencies our test has -- for example, Clang's 'check-clang'
# target would ensure 'clang' was built. We also want to ensure the googletest
# unit test C++ files are compiled into executables. Plus, as described earlier
# in this file, we certainly wish to use LLVM's FileCheck. But also, as it
# happens, lit insists some of utilities *must* exist in our build directory,
# like 'count' and 'not'.
set(EXAMPLES_TEST_DEPS
    ExamplesUnitTests
    # We call 'LLVMConfig.use_default_substitutions()' from within our lit
    # config. This function insists that the following executables can be found
    # in our build directory or in out PATH:
    FileCheck count not)

# 'add_lit_testsuite' calls through to 'add_lit_target' (both from
# AddLLVM.cmake), which calls the native CMake function 'add_custom_command',
# which creates a CMake target 'check-examples' that calls
# 'python /path/to/lit.py /path/to/build/examples/test', thereby executing our
# tests via lit. The 'DEPENDS' argument is passed along to CMake's
# 'add_dependencies' function.
add_lit_testsuite(
  check-examples "Running the examples tests" ${CMAKE_CURRENT_BINARY_DIR}
  DEPENDS ${EXAMPLES_TEST_DEPS})

# This neat function (also from 'AddLLVM.cmake') recursively searches the
# specified directory (in this case this current 'test' directory) and creates
# a CMake target (via 'add_lit_target', explained above) for each of the
# directories. For example, if you don't want to run the entire LLVM test suite
# ('check-llvm') but instead just the tests from
# 'llvm/test/Transforms/Coroutines', you can, using the
# 'check-llvm-transforms-coroutines' target created by a call to this function
# from within 'llvm/test/CMakeLists.txt'.
#
# So this function defines a target for our googletest unit tests specifically
# as well: 'check-examples-unit'.
add_lit_testsuites(EXAMPLES ${CMAKE_CURRENT_SOURCE_DIR}
  DEPENDS ${EXAMPLES_TEST_DEPS})

test-example.c

// RUN: echo "hello" | FileCheck %s
// CHECK: hello

test-lit.cfg.py

import lit.formats

from lit.llvm import llvm_config

# 'TestingConfig.name' is included in various messages printed by lit.
config.name = 'examples'

# lit has two primary test runners:
#
#   1. 'ShTest', for arbitrary files that use 'RUN:' directives.
#   2. 'GoogleTest', for C++ files that use the googletest framework.
#
# Here we use 'ShTest'. Most LLVM projects keep their googletest tests in a
# completely separate directory, such as 'llvm/unittests' (as opposed to the
# 'ShTest' tests in 'llvm/test'.
#
# The 'ShTest' constructor takes a boolean 'execute_external' parameter to
# indicate whether it should use the host machine's shell to execute 'RUN:'
# commands. This introduces a dependency between the tests and the host machine,
# and so lit also provides its own "internal" shell command executor.
#
# The LLVMConfig constructor determines whether it should use the lit shell
# executor by checking the host platform, or the presence of a
# 'LIT_USE_INTERNAL_SHELL' environment variable. By default, on macOS,
# 'use_lit_shell' is False, and so the external shell is used.
execute_external = not llvm_config.use_lit_shell
config.test_format = lit.formats.ShTest(execute_external)

# As explained in 'test/CMakeLists.txt', lit only picks up test files that
# reside in the specified 'test_source_root' directory, and end with these
# 'suffixes'.
config.test_source_root = os.path.dirname(__file__)
config.suffixes = ['.ll']

# 'LLVMConfig.use_default_substitutions' sets up 'FileCheck', 'not', and 'count'
# to be used in our test files. It also creates '%python', which points to the
# Python executable being used to invoke lit.
llvm_config.use_default_substitutions()

# We use 'LLVMConfig.use_llvm_tool' to make both the 'llc' and 'clang'
# executables available to tests in the 'examples/test' directory. This method
# can be used in two ways:
#
#   1. Ensure that the tools 'llc' and 'clang' can be found in the PATH. To do
#      so below, we append the 'LLVM_TOOLS_DIR' path that we set up in
#      'examples/test/lit.site.cfg.py.in' to the PATH.
#   2. Provide a 'search_env' argument, like this:
#
#          llvm_config.with_environment(
#              'LLC', os.path.join(config.llvm_tools_dir, 'llc'))
#          llvm_config.use_llvm_tool('llc', search_env='LLC', required=True)
#
# Appending to the PATH is simpler and doesn't require adding an environment key
# for each tool, so we use method (1) here.
llvm_config.with_environment('PATH', [config.llvm_tools_dir], append_path=True)
llvm_config.use_llvm_tool('llc', required=True)
llvm_config.use_llvm_tool('clang', required=True)

# lit allows test writers to constrain their tests to only execute in certain
# environments. For example, a test may specify that it can only run in outer
# space:
#
#     # RUN: echo "no one can hear this"
#     # REQUIRES: outer-space
#
# In this example, 'outer-space' is an arbitrary string, but it's the logic in
# our lit configuration file 'lit.cfg.py' that allows this test to be run in
# some contexts and not others. It does so by executing conditional logic and,
# in the contexts in which the test ought to be run, adding a string to
# 'TestingConfig.available_features':
#
#     if in_outer_space():
#         config.available_features.add('outer-space')
#
# Below, I'd like to allow tests to be gated by the host triple. When this
# configuration file is executed in Python as part of a lit test run on my home
# computer, this adds 'x86_64-unknown-linux-gnu'. On my laptop, it adds
# 'x86_64-apple-macosx10.14.0'. So I can specify tests to only be run on my home
# computer with 'REQUIRES: x86_64-unknown-linux-gnu', or only be run on my
# laptop with 'REQUIRES: x86_64-apple-macosx10.14.0'.
config.available_features.add(config.host_triple)

test-lit.site.cfg.py.in

# LLVM's CMake function 'configure_lit_site_cfg' will replace this with a
# comment explaining to developers that this file, which will be processed
# and copied to the build directory, is automatically re-generated when
# CMake is run to configure this project.
@LIT_SITE_CFG_IN_HEADER@

# lit loads this file via 'TestingConfig.load_from_path'. That method not only
# 'exec's this file as Python, it also sets up some variables in the context in
# which it is executed: 'config' to an instance of 'TestingConfig', and
# 'lit_config' to an instance of 'LitConfig'.
#
# In our main config file, 'lit.cfg.py', we call
# 'LLVMConfig.use_default_substitutions', which directly references a property
# 'TestingConfig.llvm_tools_dir', so we need to set this attribute on the
# 'TestingConfig' object.
#
# NB: I forgot that in Python, you can set arbitrary members on an instace of
#     a class, which is what's happening here.
config.llvm_tools_dir = "@LLVM_TOOLS_DIR@"

# 'LLVM_HOST_TRIPLE' is a string representing the host's architecture and
# operating system. In fact the term "triple", which is used widely throughout
# LLVM, is a historical artifact -- they used to contain exactly three fields,
# in the form '<architecture>-<vendor>-<operating_system>'. Now many "triples"
# include a fourth field at the end: '-<environment>'.
#
# 'LLVM_HOST_TRIPLE' is inferred automatically by LLVM's CMake function
# 'get_host_triple', defined in 'llvm/cmake/modules/GetHostTriple.cmake'. This
# function executes the shellscript 'llvm/cmake/config.guess', which in turn
# relies on the POSIX function 'uname(1)' (or at least, it does on Unix-like
# hosts) to get detailed names for the system's machine, processor, kernel
# version, and more.
#
# As an example of how the "guessing" works, here are some triples:
#
#   * 'x86_64-unknown-linux-gnu', for a 64-bit x86 machine running Ubuntu OS.
#     * 'x86_64' is the "architecture" part of the triple. It's what's returned
#       by 'uname --machine'.
#     * 'unknown' is the "vendor" here. In reality this triple is used to
#       represent a host running Ubuntu 19.04, but in practice no part of LLVM
#       relies on Linux vendors being set correctly, so 'unknown' is fine. (Not
#       so with vendors such as "apple" -- the POSIX file creation flag
#       'O_CREAT', for example, has a platform-specific value, and LLVM checks
#       whether a triple's vendor field is "apple" when determining this value.)
#     * 'linux' is the "operating system", and 'gnu' is the "environment". In
#       fact 'uname --operating-system' returns 'GNU/Linux'. This "environment"
#       field could potentially take other values, based on the host's ABI --
#       for example, 'gnueabihf' for EABI platforms that represent floating
#       point numbers in their chip hardware ("hard float").
#   * 'x86_64-apple-macosx10.14.0', for a 64-bit MacBook Pro running macOS 10.14
#      * Note that this triple does not use the "environment" field, and so is
#        truly a 3-element "triple", as per the historical naming.
#
# We set 'TestingConfig.host_triple' here so that we may use it to define tests
# that should only run on certain hosts. See 'examples/test/lit.cfg.py' for
# details.
config.host_triple = "@LLVM_HOST_TRIPLE@"

# 'lit.llvm.initialize' sets a global Python variable, 'llvm_config', set to a
# new instantiation of 'lit.llvm.config.LLVMConfig'. This makes it available
# from within our "main", project-specific config file, 'lit.cfg.py'.
#
# Among other things, the 'LLVMConfig' constructor adds 'features' depending
# on, for example, the host machine's operating system. This would allow us to
# do things like specify 'REQUIRES: system-linux' at the top of our test to
# have it run only on Linux hosts.
import lit.llvm
lit.llvm.initialize(lit_config, config)

# Finally, we have our configured file load our "main" config file, which is
# not copied to the binary directory.
lit_config.load_config(config, "@EXAMPLES_SOURCE_DIR@/test/lit.cfg.py")

test-Unit-lit.cfg.py

# This configuration file is almost identical to what's in
# 'examples/test/lit.cfg.py', but it specifies a lit "test runner" that is aware
# of how to execute googletest unit test executables.

import lit.formats

from lit.llvm import llvm_config

# We distinguish this test suite's name from the other's by appending 'Unit'.
config.name = 'examples-Unit'

# The 'GoogleTest' initializer takes two parameters:
#
#   1. A list of paths to append to the end of directories it's searching for
#      test executables. This is meant to allow Xcode and other IDEs to specify
#      a build mode, such as "Release" or "Debug", at build-time instead of at
#      CMake configuration time, and have only the unit tests built for that
#      mode be run.
#  2. A test file name suffix. lit will only exeucte unit test executables
#     whose names end in this suffix. Note that we use "Tests" as the suffix,
#     and that all test targets defined in 'examples/unittests' end in "Tests"
#     -- if any of them didn't, they would not be run.
#
#     TODO: It's my suspicion that a handful of tests in the compiler-rt and
#           lldb projects are not being run because their test name ends with
#           "Test", not "Tests". I don't normally build these targets though,
#           so I'm not going to bother to double check right now. But if it's
#           true, I think there's an argument to be made to modify
#           'add_unittest' such that it automatically appends "Tests" to the
#           target name, so test can't be accidentally named in such a way that
#           they're never run.
config.test_format = lit.formats.GoogleTest(config.llvm_build_mode, 'Tests')

# Like 'examples/test/lit.cfg.py', we need to tell lit where to find our tests
# and what their filename suffix is. In this case, we want it to recurse into
# the 'unittests' folder of our *build* directory, for googletest unit test
# executables that have no suffix (they'll be named, for example,
# 'ExamplesLLVMAnalysisTests').
config.test_source_root = os.path.join(config.examples_obj_root, 'unittests')
config.suffixes = []

# When running tests with ASan/MSan, these environment variables need to set in
# order for them to print symbolicated stack traces. The environment variables
# set when the top-level lit command is invoked isn't entirely passed through
# to the subcommands that lit run (on Windows, environment variables like 'TEMP'
# and 'TMP' are by default, but those are exceptions)-- so when lit runs each
# individial googletest, these environment variables won't be set unless we
# explicitly pass them through here.
llvm_config.with_system_environment(['ASAN_SYMBOLIZER_PATH',
                                     'MSAN_SYMBOLIZER_PATH'])

test-Unit-lit.site.cfg.py.in

# This configuration file is almost identical to what's in
# 'examples/test/lit.cfg.py.in'.

@LIT_SITE_CFG_IN_HEADER@

# TODO: Explain 'LLVM_BUILD_MODE' in detail. And test project generation with
#       Xcode.
config.llvm_build_mode = "@LLVM_BUILD_MODE@"
config.examples_obj_root = "@EXAMPLES_BINARY_DIR@"

import lit.llvm
lit.llvm.initialize(lit_config, config)

lit_config.load_config(config, "@EXAMPLES_SOURCE_DIR@/test/Unit/lit.cfg.py")

unittests-CMakeLists.txt

# The native CMake function 'add_custom_target' allows users to define an
# arbitrary target with no output. Most LLVM projects use this function to
# define an umbrella unit test target named '<Project>UnitTests': Clang defines
# 'ClangUnitTests', and lld defines 'LLDUnitTests', for example. Unit test
# binaries are then created and this umbrella target is marked dependent upon
# those. The end result is that building the 'ClangUnitTests' builds all of the
# unit tests defined as part of the Clang unit test suite.
add_custom_target(ExamplesUnitTests)
# CMake is capable of generating IDE project files, and when it does so, it
# groups source files for a target based on their 'FOLDER' value. So, for
# example, if a user were to run 'cmake -G Xcode' when configuring this project,
# the source files for all of its unit tests would be grouped together in a
# folder named "examples tests". To do so, the LLVM CMake function
# 'add_unittest' (explained in detail below) grabs this 'FOLDER' property from
# the umbrella target using the native CMake function 'get_target_property'.
set_target_properties(ExamplesUnitTests PROPERTIES FOLDER "examples tests")

# Most LLVM projects define a wrapper function like this one around LLVM's
# 'add_unittest'. (Note: Don't be fooled! Although most LLVM CMake functions
# include 'llvm' in their names, 'add_unittest' does not. It is *not* a native
# CMake function, and in fact it doesn't even call through to the native CMake
# function 'add_test'.)
#
# LLVM's 'add_unittest' takes two positional parameters: the test suite name
# ('ExamplesUnitTests' in this case) and the name of the test (for example,
# 'ExamplesLLVMAnalysisTests'). Any number of source files can be passed in
# after that, and 'add_unittest' will call through to 'add_llvm_executable'
# (explained in detail in the 'modocache/cl' project, in
# 'cl/cmake/modules/AddCl.cmake') to create an executable with the given name,
# composed of the one or more source files. 'add_unittest' then marks this
# executable as a dependency upon the umbrella test target -- in this case,
# that's 'ExamplesUnitTests'. It also handles things like respecting the user's
# 'LLVM_BUILD_TESTS' settings (explained in detail in
# 'examples/CMakeLists.txt'), addding the googletest headers to the target's
# include paths, and linking the executable against googletest and the
# 'gtest_main' library. As explained further in
# 'examples/unittests/Example.cpp', this is a library that defines a 'main'
# function that runs all the test cases defined in the executable.
#
# Note that using this function just defines a target to build an executable
# that, when run, executes the test. It's the lit googletest test runner that
# actually then executes the test executable. Read the comments in
# 'examples/test/CMakeLists.txt' and 'examples/test/Unit' for details.
function(add_examples_unittest name)
  add_unittest(ExamplesUnitTests ${name} ${ARGN})
endfunction()

# Here we call the function we defined above to define an actual test target.
# Normally, we'd need to link in LLVMSupport by writing something like
# 'set(LLVM_LINK_COMPONENTS Support)', but the underlying LLVM CMake function
# 'add_unittest' takes care of this for us. This is because LLVM actually uses a
# fork of googletest, which lives in 'llvm/utils/unittest/googletest'. This
# version of googletest is based on googletest 1.8.0, but modified to use LLVM
# abstractions like LLVMSupport's 'llvm::raw_ostream'.
add_examples_unittest(ExamplesTests
                      ExampleTest.cpp)

unittests-ExampleTest.cpp

#include "gtest/gtest.h"

// Tests defined with the googletest framework are simply executables that are
// compiled from C++ source code. Like most C++ executables, their entry point
// is the 'main' function. So, how do we get from the simple 'TEST(...) { ... }'
// defined below, to an executable that executes the expectation
// 'EXPECT_EQ(2, 1 + 1)'?
//
// The googletest 'TEST' macro expands several other macros, which eventually
// expand this line into a declaration, of the class
// 'ExampleTest_onePlusOneIsTwo_Test', which subclasses an abstract base class
// 'testing::TestCase'. This base class defines a 'Run' method, which in turn
// executes the 'TestBody'. Down below, the braces '{ ... }' that users write
// following the use of the 'TEST' macro are expanded into a definition of
// 'ExampleTest_onePlusOneIsTwo_Test::TestBody'.
//
// In addition, the macro expands to declare and initialize
// the member 'ExampleTest_onePlusOneIsTwo_Test::test_info_'. This member is
// initialized via a call to 'testing::internal::MakeAndRegisterTestInfo', which
// eventually calls 'testing::internal::UnitTestImpl::GetTestCase'.
// 'testing::UnitTest' and its 'testing::internal::UnitTestImpl' are lazily
// instantiated singletons that store a 'std::vector<TestCase *>'. The 'TEST'
// macro expands to declate a 'TestCase' class, instantiate it, and push it onto
// this 'UnitTestImpl' singleton's vector of test cases, via a call to
// 'GetTestCase'.
//
// The last piece of the puzzle is the 'main' function -- what's first executed
// when the executable build from this source code is run. The googletest
// framework provides two ways for users to write this 'main' function:
//
//   1. Define their own 'main' function, which must include the following lines
//      of code:
//
//          int main(int argc, char **argv) {
//            testing::InitGoogleTest(argc, argv);
//            return RUN_ALL_TESTS();
//          }
//   2. Link their executable against 'gtest_main', which defines such a 'main'
//      function.
//
// The 'RUN_ALL_TESTS' function calls through to the singleton
// 'testing::UnitTest' method 'Run', which parses command-line arguments and,
// based on any filters defined in those arguments, runs some or all of the
// tests stored in its test cases vector.
TEST(ExampleTest, onePlusOneIsTwo) { EXPECT_EQ(2, 1 + 1); }