outdoordoor

Exploring macro-free testing in modern C++

Some time ago I wrote a blogpost about a basic C++ unit-testing library I made that aimed to use no macros, unlike many established frameworks like gtest. I got some great feedback after that and decided to improve the library and release it as a standalone project. It's not intended to stand up to the giants, but is more of a fun little experiment on what a library like this could look like. Enter nmtest.

Table of contents:

Why no macros?

Before diving deeper into the implementation, I'd like to answer the question I got a couple of times: why avoid macros? And that's a valid thing to ask: as we'll see later, some tasks are quite hard to do even in modern C++ without them.

To be honest, the first and main reason is that this was a self-imposed challenge I did for fun. I saw that many major testing frameworks are very macro-heavy and wanted to see if I could do without them.

However, there are also several objective reasons to avoid macros in general. They bypass type-checking, make you rely on your IDE too much, and are a pain to debug. Moreover, the C++20 modules encourage reducing the use of macros by prohibiting their export.

nmtest

Now let's get back to the library itself. The list of changes I wanted to make since the last post included:

Such libraries often include other things like test order shuffling, timing, etc., but I decided to focus on the things listed above for now.

General structure

Improvements to the structure began with restructuring of the code file. I wanted to still have the whole library in one file, so I separated its contents into several namespaces:

Since I'm using C++20 modules, I could export only the nm namespace and keep the others hidden.

The structure of the test storage was also changed. First of all, the ability to create custom Registry objects was removed. Besides being redundant (generally, you need one registry per project), creating custom registries could be a serious obstacle to automatic test registration.

So, now there's only one global registry that stores suites and tests in a tree-like structure. I opted for the tree and not the flat-storage-with-indices structure, since it's generally easier to reason about as a developer, even though flat storage is generally more performant for filtering.

Cleaning up test functions

Another improvement I wanted to include is one to the test functions. Previously, they returned the Report structure to collect the information on different checks and asserts. However, that meant, that even for the simplest tests you had to write

return Report{} & Equal(1, 2);

This was quite inconvenient, so I decided to move the storage of fail info from the Report to the Result structure, which all built-in functions already return. This means that you now don't need Report at all, and can just write:

return Equal(1, 2);

while preserving all the information on test failures like before. As for chaining, it still works like before, allowing any number of checks be performed in the same test:

return
  Equal   (3, 5)
& Equal   (1, 1)
& NotEqual(2, 2, "custom message"); // oh yeah, and custom error 
                                    // messages are now also supported

The test result output became a bit more concise, but looks similar to the previous version:

[   PASS] Addition
[X  FAIL] Subtraction:
          - Equal(1:5) | main.cpp:146
          - NotEqual(2:2) | main.cpp:148 | custom message

Automatic test discovery

The obvious solution (and why it's not enough)

This was the most interesting part of the project, even though at first it seems quite simple. Since in this project I'm mainly experimenting, I've left several ways to create a test available in the API, just as a demonstration of how it could be implemented.

If a test is created in a function, then automatically registering it poses no challenge: you can just make a function that receives all test info through the parameters, and then register it in the body. Alternatively, you can choose to make an object that registers the test in the constructor. This was the first implementation I added, although I did break it down into several methods a user can chain:

Test      ("suite name", "test name")
.Setup    ([]{ /* optional setup function */ })
.Teardown ([]{ /* optional teardown function */ })
.Tags     ({ /* optional tag list */ })
.Func     ([]{ return Equal(1, 2); });

This creates only one test, but making it work for a whole suite was not difficult either. Although here I went for using the designated initialization when registering the test.

Suite("suite name")
.Setup([]{ /* optional suite setup*/ })
.Teardown([]{ /* optional suite teardown*/ })
.Test({
    .name = "test name 1",
    .func = []{ return Equal(0,9); }
})
.Test({
    .name = "test name 2",
    .func = []{ return Equal(0,8); }
});

Both functions are similar internally, they access the static global registry, and add the suite or test. This works, but has a major limitation - you can't use these methods in the global scope outside functions, code like this is not allowed in C++:

Test("suite name", "test name")
.Func([]{ return Equal(2+2, 4); })

int main() { Run(); }
Global scope

One way to circumvent this is to create actual objects and register them through static initialization. Since it occurs before the main() is called, it's safe to assume that all tests are registered before the call to Run().

For this purpose I implemented the TestS struct. There was one problem I had to solve: static initialization implies having a custom constructor that does the registration, but having a user-declared constructor makes the type non-aggregate which disallows designated initialization (which I wanted to keep).

However, aggregates can still have members that are not aggregates. So, to go around the limitation, I added a number of structs looking like this:

struct SuiteName
{
    SuiteName(const std::string& name)
    {
        if (!name.empty()) GetRegistry().LastSuite(&(GetRegistry().GetSuite(name)));
        else GetRegistry().LastSuite(nullptr);
    }
    SuiteName(const char* name) : SuiteName(std::string(name)) {}
};
struct TestName
{
    TestName(const std::string& name)
    {
        if (!name.empty() && GetRegistry().LastSuite())
            GetRegistry().LastTest(&(GetRegistry().LastSuite()->Test(name)));
        else GetRegistry().LastTest(nullptr);
    }
    TestName(const char* name) : TestName(std::string(name)) {}
};

Now, instead of having standard strings or functions in the TestS, I have these custom types that do the registration. This allows to simplify the TestS struct to an aggregate:

struct TestS
{
    impl::SuiteName    suite;
    impl::TestName     name;
    impl::TestFunc     func;
    impl::TagList      tags;
    impl::SetupFunc    setup;
    impl::TeardownFunc teardown;
};

// now with designated initialization!
TestS t{
    .suite = "suite name",
    .name  = "test name",
    .func  = []{ return Equal(10,20); },
};

Since each element registers only itself, I had to make sure all elements of one test go into the same test. For this I added the lastSuite and lastTest pointers to the registry, which are updated when a new suite and test are created accordingly. Since member initialization goes in the order of their declaration, it's safe to assume these pointers point where they should.

As you may have noticed from the example above, there's still one major drawback to this method - you have to manually choose a name for each test object (t in this case), since you can't just create a temporary object in the global scope.

And you can't just somehow generate these names, even template magic won't help. Usually, it's where you turn to macros, after all, code generation is their meat and potatoes. But remember, the whole goal of this project is to use no macros, so we need to look elsewhere.

Templates to the rescue

Even if we can't generate names with templates, maybe we could still use them for test registration? And it turns out - yes, we can. The following method was suggested to me by the user u/triconsonantal, for which I'm grateful. They actually suggested two methods, here's the first one that I ended up using, and the second one.

As a sneak-peak, here's what the code allows us to do:

// no object name needed!
template class TestT<
  "suite name",
  "test name",
  []{ return Equal(1,2); }>;

The core concept it uses is NTTP (non-type template parameters). In its simplest form you could encounter it in, for example, standard arrays, where you may write:

std::array<int, 3> myArray; // '3' is not a type

However, we could expand this for our case. First, we need to prepare suitable types for the parameters. NTTP does not allow normal strings, since they are not structural. So, as a replacement, we can create a custom implementation:

template<std::size_t N>
struct StructuralString
{
    static_assert(N > 0);

    constexpr StructuralString(const char (&str)[N])
    {
        std::copy_n(str, N, string);
    }

    constexpr operator std::string_view () const { return string; }
    constexpr operator std::string () const { return string; }

    char string[N] {};
};

This covers test and suite names, but we still need tag lists and several functions. For the former we can just use an std::array, and for the latter we use function pointers: Result (* func) () and void (* func) () for test and setup/teardown functions accordingly.

Functions, however, are where we see first limitations of this approach: lambdas that we use as values for those parameters cannot capture anything, otherwise they'd be not a structural type.

With suitable types in place we can start implementing the structure I named TestT:

template<impl::StructuralString suite,
         impl::StructuralString name,
         Result (* func) (),
         auto tags = std::array<const char*, 0>{},
         void (* setup) () = {},
         void (* teardown) () = {}>
class TestT
{
    TestT()
    {
        // test registration like before
    }

    static const TestT registerer;
};

Now we only need to force template instantiation that would trigger the constructor and register our test. For this we have the static registerer object that calls the private constructor, which ensures that for each instantiation of the template there's only one registration.

Since we need to not only declare the static member, but to also define it, outside the class we write:

template<impl::StructuralString suite,
             impl::StructuralString name,
             Result (* func) (),
             auto tags = std::array<const char*, 0>{},
             void (* setup) () = nullptr,
             void (* teardown) () = nullptr>
    const TestT<suite, name, func, tags, setup, teardown> TestT<suite, name, func, tags, setup, teardown>::registerer;

Now, each time we write

template class TestT<{ ... }>;

we instantiate a new type, forcing the constructor to be called and registering our test.

One more issue that remains is that currently, test creation cannot be in headers, since including such headers in multiple translation units would create duplicates of the same test. To fix this, we add

template<impl::StructuralString suite,
         impl::StructuralString name>
inline bool testRegistered = false;

outside the class and check it in the constructor. This ensures that there can be only one test with the same suite-name combination.

A single full test with such approach could look like this:

template class TestT<
  "suite name",
  "test name",
  []{ return Equal(1,2); },
  std::array{"tag 1", "tag 2"},
  []{ std::println("setup"); },
  []{ std::println("teardown"); }>;

which, while not as clean as I would've liked, still looks fine given the restrictions.

Test filtering and CLI

As could be seen from the previous examples, tests can be filtered not just by suites, but by tags as well. They are stored as a simple array of strings and are checked at runtime against the filters provided through the CLI.

It was fun to learn more about the standard views library, to which I had quite limited exposure in the past. In the case of filters, it allowed me to write code like this:

for (const auto& [testName, test] : suite.Tests()
    | std::views::filter([&](const auto& pair)
    {
        // actual filtering
    }))

As for the CLI, I went for a basic solution for now. It allows running all tests or specifying which suites and tags to run. For example, this command runs all tests from suites "core" or "math" that have tag "fast":

./test -s "core,math" -t "fast"

I've also added a couple of options like --verbose for detailed output, --list for listing tests that match the filters without running them, --case_sensitive to enable case sensitivity for identifiers, and of course --help.

Conclusion

All in all, this project was an interesting experiment to see how unit-testing could be implemented without any macros. It's obvious that C++ still doesn't give us all the tools needed to fully get rid of macros, but even now we can work around these limitations to achieve similar functionality.