There are two main kinds of automated testing supported by Haskell testing libraries:
-
Unit tests, where some data is generally hardcoded. Generally, each test case will be designed to cover a class of possible inputs. They explicitly cover many cases, but if the tester forgets some corner case, it will certainly not be covered.
-
Property tests, where some data is randomly generated for the tests. This has no guarantee of covering all classes of possible inputs, but there is a large chance of capturing unforeseen corner cases.
The libraries available for writing tests are the following.
-
HUnit helps with writing unit in the spirit of
JUnit. -
HSpec integrates with HUnit and provides text-based combinators such as
shouldBeinstead of@=?. -
QuickCheck provides helpers for generating random input data and defining property tests.
-
SmallCheck is similar to QuickCheck but focuses on exhaustively checking all "small" instances of a type. This may not scale well if a type has too many small instances
A large project will contain several test cases grouped into several test suites. There are many Haskell libraries that provide ways to organize such tests, in two main styles.
- Explicit suite management, where lists or trees of test suites are written in the Haskell source files.
- Test discovery, where some framework traverses the Haskell source files collecting the existing tests.
In the following subsections we discuss the available options. Note that QuickCheck and SmallCheck don't provide any utilities for organizing tests.
HUnit allows explicit management, but it is not the focus of this library. The data constructors TestList and TestLabel allow tests to be grouped. Some operators like ~: and list are provided to try and make it more readable. Those who dislike too many operators will find it unpleasant.
Furthermore, this only allows HUnit tests to be grouped, so integration with property testing is difficult.
HSpec allows explicit management, and its extension hspec-discover provides test discovery. It integrates with QuickCheck, SmallCheck and HUnit, and allows parallel test execution. It has pretty good documentation.
The main focus of HSpec is test management. It provides a DSL for describing test suites, and allows integration with property testing, as may be seen in this example.
Its test discovery tools, hspec-discover will scan the filesystem to find spec files, which must end in Spec.hs should export a variable spec :: Spec, as explained in its documentation.
Tasty is a test organization tool that integrates with many other utilities including HUnit, HSpec, QuickCheck and SmallCheck. It also has an extension tasty-discover providing test discovery. It allows parallel test execution and may generate different outputs such as HTML or CSV reports. It also has very good documentation.
For manual test management, tasty allows the user to define trees of test cases as shown in this example.
Its test discovery tool will scan the files ending in Test.hs (or some other suffix, it's configurable). It will gather for top-level definitions such as:
- QuickCheck properties
prop_* :: QuickCheck.Property, - SmallCheck properties
scprop_* :: SmallCheck.Property, - HUnit test cases
unit_* :: HUnit.Test, - HSpec specifications
spec_* :: Spec, - Tasty test trees
test_* :: TestTree.
test-framework is an older, less maintained utility for manual test management similar to tasty.
HTF is a test discovery tool for QuickCheck and HUnit tests. Older, less documented and requires every test module to begin with the following lines.
{-# OPTIONS_GHC -F -pgmF htfpp #-}
module Foo ( {- other exports... -}, htf_thisModuleTests ) where