dynarmic/docs/test-cases-and-sections.md

Test cases and sections

Contents
Tags
Tag aliases
BDD-style test cases
Type parametrised test cases
Signature based parametrised test cases

While Catch fully supports the traditional, xUnit, style of class-based fixtures containing test case methods this is not the preferred style.

Instead Catch provides a powerful mechanism for nesting test case sections within a test case. For a more detailed discussion see the tutorial.

Test cases and sections are very easy to use in practice:

• TEST_CASE( test name [, tags ] )
• SECTION( section name, [, section description ] )

test name and section name are free form, quoted, strings. The optional tags argument is a quoted string containing one or more tags enclosed in square brackets, and are discussed below. section description can be used to provide long form description of a section while keeping the section name short for use with the -c command line parameter.

Test names must be unique within the Catch executable.

For examples see the Tutorial

Tags

Tags allow an arbitrary number of additional strings to be associated with a test case. Test cases can be selected (for running, or just for listing) by tag - or even by an expression that combines several tags. At their most basic level they provide a simple way to group several related tests together.

As an example - given the following test cases:

TEST_CASE( "A", "[widget]" ) { /* ... */ }
TEST_CASE( "B", "[widget]" ) { /* ... */ }
TEST_CASE( "C", "[gadget]" ) { /* ... */ }
TEST_CASE( "D", "[widget][gadget]" ) { /* ... */ }

The tag expression, "[widget]" selects A, B & D. "[gadget]" selects C & D. "[widget][gadget]" selects just D and "[widget],[gadget]" selects all four test cases.

For more detail on command line selection see the command line docs

Tag names are not case sensitive and can contain any ASCII characters. This means that tags [tag with spaces] and [I said "good day"] are both allowed tags and can be filtered on. However, escapes are not supported however and [\]] is not a valid tag.

The same tag can be specified multiple times for a single test case, but only one of the instances of identical tags will be kept. Which one is kept is functionally random.

Special Tags

All tag names beginning with non-alphanumeric characters are reserved by Catch. Catch defines a number of "special" tags, which have meaning to the test runner itself. These special tags all begin with a symbol character. Following is a list of currently defined special tags and their meanings.

• [.] - causes test cases to be skipped from the default list (i.e. when no test cases have been explicitly selected through tag expressions or name wildcards). The hide tag is often combined with another, user, tag (for example [.][integration] - so all integration tests are excluded from the default run but can be run by passing [integration] on the command line). As a short-cut you can combine these by simply prefixing your user tag with a . - e.g. [.integration].

• [!throws] - lets Catch know that this test is likely to throw an exception even if successful. This causes the test to be excluded when running with -e or --nothrow.

• [!mayfail] - doesn't fail the test if any given assertion fails (but still reports it). This can be useful to flag a work-in-progress, or a known issue that you don't want to immediately fix but still want to track in your tests.

• [!shouldfail] - like [!mayfail] but fails the test if it passes. This can be useful if you want to be notified of accidental, or third-party, fixes.

• [!nonportable] - Indicates that behaviour may vary between platforms or compilers.

• [#<filename>] - running with -# or --filenames-as-tags causes Catch to add the filename, prefixed with # (and with any extension stripped), as a tag to all contained tests, e.g. tests in testfile.cpp would all be tagged [#testfile].

• [@<alias>] - tag aliases all begin with @ (see below).

• [!benchmark] - this test case is actually a benchmark. Currently this only serves to hide the test case by default, to avoid the execution time costs.

Tag aliases

Between tag expressions and wildcarded test names (as well as combinations of the two) quite complex patterns can be constructed to direct which test cases are run. If a complex pattern is used often it is convenient to be able to create an alias for the expression. This can be done, in code, using the following form:

CATCH_REGISTER_TAG_ALIAS( <alias string>, <tag expression> )

Aliases must begin with the @ character. An example of a tag alias is:

CATCH_REGISTER_TAG_ALIAS( "[@nhf]", "[failing]~[.]" )

Now when [@nhf] is used on the command line this matches all tests that are tagged [failing], but which are not also hidden.

BDD-style test cases

In addition to Catch's take on the classic style of test cases, Catch supports an alternative syntax that allow tests to be written as "executable specifications" (one of the early goals of Behaviour Driven Development). This set of macros map on to TEST_CASEs and SECTIONs, with a little internal support to make them smoother to work with.

• SCENARIO( scenario name [, tags ] )

This macro maps onto TEST_CASE and works in the same way, except that the test case name will be prefixed by "Scenario: "

• GIVEN( something )
• WHEN( something )
• THEN( something )

These macros map onto SECTIONs except that the section names are the something texts prefixed by "given: ", "when: " or "then: " respectively. These macros also map onto the AAA or A³ test pattern (standing either for Assemble-Activate-Assert or Arrange-Act-Assert), and in this context, the macros provide both code documentation and reporting of these parts of a test case without the need for extra comments or code to do so.

Semantically, a GIVEN clause may have multiple independent WHEN clauses within it. This allows a test to have, e.g., one set of "given" objects and multiple subtests using those objects in various ways in each of the WHEN clauses without repeating the initialisation from the GIVEN clause. When there are dependent clauses -- such as a second WHEN clause that should only happen after the previous WHEN clause has been executed and validated -- there are additional macros starting with AND_:

• AND_GIVEN( something )
• AND_WHEN( something )
• AND_THEN( something )

These are used to chain GIVENs, WHENs and THENs together. The AND_* clause is placed inside the clause on which it depends. There can be multiple independent clauses that are all dependent on a single outer clause.

SCENARIO( "vector can be sized and resized" ) {
    GIVEN( "An empty vector" ) {
        auto v = std::vector<std::string>{};

        // Validate assumption of the GIVEN clause
        THEN( "The size and capacity start at 0" ) {
            REQUIRE( v.size() == 0 );
            REQUIRE( v.capacity() == 0 );
        }

        // Validate one use case for the GIVEN object
        WHEN( "push_back() is called" ) {
            v.push_back("hullo");

            THEN( "The size changes" ) {
                REQUIRE( v.size() == 1 );
                REQUIRE( v.capacity() >= 1 );
            }
        }
    }
}

This code will result in two runs through the scenario:

Scenario : vector can be sized and resized
  Given  : An empty vector
  Then   : The size and capacity start at 0

Scenario : vector can be sized and resized
  Given  : An empty vector
  When   : push_back() is called
  Then   : The size changes

See also runnable example on godbolt, with a more complicated (and failing) example.

AND_GIVEN was introduced in Catch2 2.4.0.

When any of these macros are used the console reporter recognises them and formats the test case header such that the Givens, Whens and Thens are aligned to aid readability.

Other than the additional prefixes and the formatting in the console reporter these macros behave exactly as TEST_CASEs and SECTIONs. As such there is nothing enforcing the correct sequencing of these macros - that's up to the programmer!

Type parametrised test cases

In addition to TEST_CASEs, Catch2 also supports test cases parametrised by types, in the form of TEMPLATE_TEST_CASE, TEMPLATE_PRODUCT_TEST_CASE and TEMPLATE_LIST_TEST_CASE.

• TEMPLATE_TEST_CASE( test name , tags, type1, type2, ..., typen )

Introduced in Catch2 2.5.0.

test name and tag are exactly the same as they are in TEST_CASE, with the difference that the tag string must be provided (however, it can be empty). type1 through typen is the list of types for which this test case should run, and, inside the test code, the current type is available as the TestType type.

Because of limitations of the C++ preprocessor, if you want to specify a type with multiple template parameters, you need to enclose it in parentheses, e.g. std::map<int, std::string> needs to be passed as (std::map<int, std::string>).

Example:

TEMPLATE_TEST_CASE( "vectors can be sized and resized", "[vector][template]", int, std::string, (std::tuple<int,float>) ) {

    std::vector<TestType> v( 5 );

    REQUIRE( v.size() == 5 );
    REQUIRE( v.capacity() >= 5 );

    SECTION( "resizing bigger changes size and capacity" ) {
        v.resize( 10 );

        REQUIRE( v.size() == 10 );
        REQUIRE( v.capacity() >= 10 );
    }
    SECTION( "resizing smaller changes size but not capacity" ) {
        v.resize( 0 );

        REQUIRE( v.size() == 0 );
        REQUIRE( v.capacity() >= 5 );

        SECTION( "We can use the 'swap trick' to reset the capacity" ) {
            std::vector<TestType> empty;
            empty.swap( v );

            REQUIRE( v.capacity() == 0 );
        }
    }
    SECTION( "reserving smaller does not change size or capacity" ) {
        v.reserve( 0 );

        REQUIRE( v.size() == 5 );
        REQUIRE( v.capacity() >= 5 );
    }
}

• TEMPLATE_PRODUCT_TEST_CASE( test name , tags, (template-type1, template-type2, ..., template-typen), (template-arg1, template-arg2, ..., template-argm) )

Introduced in Catch2 2.6.0.

template-type1 through template-typen is list of template template types which should be combined with each of template-arg1 through template-argm, resulting in n * m test cases. Inside the test case, the resulting type is available under the name of TestType.

To specify more than 1 type as a single template-type or template-arg, you must enclose the types in an additional set of parentheses, e.g. ((int, float), (char, double)) specifies 2 template-args, each consisting of 2 concrete types (int, float and char, double respectively). You can also omit the outer set of parentheses if you specify only one type as the full set of either the template-types, or the template-args.

Example:

template< typename T>
struct Foo {
    size_t size() {
        return 0;
    }
};

TEMPLATE_PRODUCT_TEST_CASE("A Template product test case", "[template][product]", (std::vector, Foo), (int, float)) {
    TestType x;
    REQUIRE(x.size() == 0);
}

You can also have different arities in the template-arg packs:

TEMPLATE_PRODUCT_TEST_CASE("Product with differing arities", "[template][product]", std::tuple, (int, (int, double), (int, double, float))) {
    TestType x;
    REQUIRE(std::tuple_size<TestType>::value >= 1);
}

• TEMPLATE_LIST_TEST_CASE( test name, tags, type list )

Introduced in Catch2 2.9.0.

type list is a generic list of types on which test case should be instantiated. List can be std::tuple, boost::mpl::list, boost::mp11::mp_list or anything with template <typename...> signature.

This allows you to reuse the type list in multiple test cases.

Example:

using MyTypes = std::tuple<int, char, float>;
TEMPLATE_LIST_TEST_CASE("Template test case with test types specified inside std::tuple", "[template][list]", MyTypes)
{
    REQUIRE(sizeof(TestType) > 0);
}

Signature based parametrised test cases

Introduced in Catch2 2.8.0.

In addition to type parametrised test cases Catch2 also supports signature base parametrised test cases, in form of TEMPLATE_TEST_CASE_SIG and TEMPLATE_PRODUCT_TEST_CASE_SIG. These test cases have similar syntax like type parametrised test cases, with one additional positional argument which specifies the signature.

Signature

Signature has some strict rules for these tests cases to work properly:

• signature with multiple template parameters e.g. typename T, size_t S must have this format in test case declaration ((typename T, size_t S), T, S)
• signature with variadic template arguments e.g. typename T, size_t S, typename...Ts must have this format in test case declaration ((typename T, size_t S, typename...Ts), T, S, Ts...)
• signature with single non type template parameter e.g. int V must have this format in test case declaration ((int V), V)
• signature with single type template parameter e.g. typename T should not be used as it is in fact TEMPLATE_TEST_CASE

Currently Catch2 support up to 11 template parameters in signature

Examples

• TEMPLATE_TEST_CASE_SIG( test name , tags, signature, type1, type2, ..., typen )

Inside TEMPLATE_TEST_CASE_SIG test case you can use the names of template parameters as defined in signature.

TEMPLATE_TEST_CASE_SIG("TemplateTestSig: arrays can be created from NTTP arguments", "[vector][template][nttp]",
  ((typename T, int V), T, V), (int,5), (float,4), (std::string,15), ((std::tuple<int, float>), 6)) {

    std::array<T, V> v;
    REQUIRE(v.size() > 1);
}

• TEMPLATE_PRODUCT_TEST_CASE_SIG( test name , tags, signature, (template-type1, template-type2, ..., template-typen), (template-arg1, template-arg2, ..., template-argm) )

template<typename T, size_t S>
struct Bar {
    size_t size() { return S; }
};

TEMPLATE_PRODUCT_TEST_CASE_SIG("A Template product test case with array signature", "[template][product][nttp]", ((typename T, size_t S), T, S), (std::array, Bar), ((int, 9), (float, 42))) {
    TestType x;
    REQUIRE(x.size() > 0);
}

---

Home