outdoordoor

Macro-free unit-testing framework in modern C++

For my current project I wanted to have a small testing library, and since I find joy in reinventing the wheel, I decided to write my own.

While many testing frameworks rely extensively on macros in their APIs, using the preprocessor too much always felt wrong to me. Bypassing the type system and confusing refactoring tools, they make you rely on your IDE and your own memory a bit too much for my taste.

So my main goal was to make an API that would be as easy to use as a macro-based one, but without its downsides. I also went for a declarative style by allowing to define the structure of tests through method chaining.

API use example:

Registry regs;
regs.Suite("First suite")
    .Setup([]{ /* optional suite setup */ })
    .Teardown([]{ /* optional suite teardown */ })
    .Add(
        Test("First test")
        .Setup([]{ /* optional test setup */ })
        .Teardown([]{ /* optional test teardown */ })
        .Func([]
        {
            return Report {} 
            & Equal(1, 1) 
            & Equal(1, 2) 
            & Equal(1.2, 1.4) 
            & Throws([] { throw "a"; }) 
            & Throws([] {}) 
            & DoesNotThrow([] {}) 
            & DoesNotThrow([] { throw "a"; });
        })
);

regs.Run();

Which outputs:

[  SUITE] First suite
[X  FAIL] First test :
          Failed assert: 'Equal [1:2]'; File: 'D:\testlib\src\main.cpp'; Line: '25'
          Failed assert: 'Equal [1.2:1.4]'; File: 'D:\testlib\src\main.cpp'; Line: '26'
          Failed assert: 'Throws'; File: 'D:\testlib\src\main.cpp'; Line: '24'
          Failed assert: 'DoesNotThrow'; File: 'D:\testlib\src\main.cpp'; Line: '26'
[SUMMARY] Total: 1; Succeeded: 0; Failed: 1
          Failed tests:
          - First test

Library design

The general hierarchy is the following:

Let's look at these from the bottom up.

Assert

An assert is a single check represented by a function. Here's an example of an assert:

// succeeds if the actual value is equal to the expected value
// works for floating point numbers as well
template<typename T>
[[nodiscard]]
constexpr auto Equal(const T& actual, const T& expected, const std::source_location loc = std::source_location::current()) -> Result
{
    const auto res = (types::Number<T>) ? math::NumericEqual(actual, expected) : (actual == expected);
    return res ? Result(true) : Result(false, std::format("Equal [{}:{}]", actual, expected), loc);
}

You can notice that it returns some Result type. This is a structure that contains a bool result of the assert and a string message used later for output. It contains the type of the assert ("Equal" in this case), actual and expected values (where applicable), and the source location of the assert. The message is formed only if the assert failed.

The library offers several types of asserts - (Not)Equal, (Not)Null, True, False, Throws, DoesNotThrow - and the list can be easily expanded.

Test

A test is a function that contains a number of asserts. The simplest way to create a test is with a lambda, as shown in the example above. Tests can have optional Setup and Teardown functions that run before and after the test accordingly.

Tests need to aggregate the results of the asserts and report them back to the Registry. Since assert results are not a single bool but a structure with an optional message, the way of collecting all these messages is more complex. For this purpose, the library uses the Report structure, containing:

// the combined result of the whole test
bool success = true;
// a list of all messages
std::vector<std::string> messages;

It also provides a convenience function for formatting the report and an overload for the & operator, which enables convenient assert chaining shown in the examples.

Suite

A suite can be used for grouping related tests together. It also provides the ability to add Setup and Teardown functions. To enable convenient test registration, the Suite class provides a variadic template function Add, which allows adding any number of tests in a single call:

template<typename T, typename... Ts>
requires (std::same_as<std::decay_t<T>, Test> &&
         (std::same_as<std::decay_t<Ts>, std::decay_t<T>> && ...))
auto Add(T&& first, Ts&&... rest) -> void
{
    tests.emplace_back(std::forward<T>(first));
    (tests.emplace_back(std::forward<Ts>(rest)), ...);
}

Registry

And finally, the structure that contains all the test suites. Its main purpose is storing the suites and running them in a controlled manner. The registry outputs all the Reports from tests and keeps track of the number of total and failed tests. It also handles any unhandled exceptions that may occur inside the test functions and reports them, so this test:

Test("Exceptional test")
.Func([]
{
    throw std::runtime_error("some exception");
    return Report{};
})

produces this output:

[  SUITE] First suite
[! ERROR] Test 'Exceptional test': Test function has thrown an unhandled exception 'some exception'
[SUMMARY] Total: 0; Succeeded: 0; Failed: 0; Errors: 1

The same goes for any unhandled exceptions in the setup or teardown functions.

Conclusion

For now I'm using the library in a personal project as a part of the built-in testing environment. It has worked quite well for me so far. I've not yet released it as a standalone project, but you can find it here in the test module (note that it also depends on the math and types modules).

Some time ago I also released the dough library, where I tried to achieve similar goals. I've improved in some areas since then, but dough offers more features like tagging and CLI, which I might later add to the current library.