Published by marco on
These days nobody who’s anybody in the software-development world is writing software without tests. Just writing them doesn’t help make the software better, though. You also need to be able to execute tests—reliably and quickly and repeatably.
That said, you’ll have to get yourself a test runner, which is a different tool from the compiler or the runtime. That is, just because your tests compile (satisfy all of the language rules) and could be executed doesn’t mean that you’re done writing them yet.
Every testing framework has its own rules for how the test runner selects methods for execution as tests. The standard configuration options are:
Each testing framework will offer different ways of configuring your code so that the test runner can find and execute setup/test/teardown code. To write NUnit tests, you decorate classes, methods and parameters with C# attributes.
The standard scenario is relatively easy to execute—run all methods with a Test
attribute in a class with a TestFixture
attribute on it.
There are legitimate questions for which even the best specification does not provide answers.
When you consider multiple base classes and generic type arguments, each of which may also have NUnit attributes, things get a bit less clear. In that case, not only do you have to know what NUnit offers as possibilities but also whether the test runner that you’re using also understands and implements the NUnit specification in the same way. Not only that, but there are legitimate questions for which even the best specification does not provide answers.
At Encodo, we use Visual Studio 2015 with ReSharper 9.2 and we use the ReSharper test runner. We’re still looking into using the built-in VS test runner—the continuous-testing integration in the editor is intriguing[1]—but it’s quite weak when compared to the ReSharper one.
So, not only do we have to consider what the NUnit documentation says is possible, but we must also know what how the R# test runner interprets the NUnit attributes and what is supported.
Where is there room for misunderstanding? A few examples,
TestFixture
attribute on an abstract class?TestFixture
attribute on a class with generic parameters?Tests
but no TestFixture
attribute?Tests
but no TestFixture
attribute, but there are non-abstract descendants that do have a TestFixture
attribute?In our case, the answer to these questions depends on which version of R# you’re using. Even though it feels like you configured everything correctly and it logically should work, the test runner sometimes disagrees.
Throw the TeamCity test runner into the mix—which is ostensibly the same as that from R# but still subtly different—and you’ll have even more fun.
At any rate, now that you know the general issue, I’d like to share how the ground rules we’ve come up with that avoid all of the issues described above. The text below comes from the issue I created for the impending release of Quino 2.
Non-leaf-node base classes should never appear as nodes in test runners. A user should be able to run tests in descendants directly from a fixture or test in the base class.
Non-leaf-node base classes are shown in the R# test runner in both versions 9 and 10. A user must navigate to the descendant to run a test. The user can no longer run all descendants or a single descendant directly from the test.
Relatively recently, in order to better test a misbehaving test runner and accurately report issues to JetBrains, I standardized all tests to the same pattern:
TestFixture
attribute only on leaf nodesThis worked just fine with ReSharper 8.x but causes strange behavior in both R# 9.x and 10.x. We discovered recently that not only did the test runner act strangely (something that they might fix), but also that the unit-testing integration in the files themselves behaved differently when the base class is abstract (something JetBrains is unlikely to fix).
You can see that R# treats a non-abstract class with tests as a testable entity, even when it doesn’t actually have a TestFixture
attribute and even expects a generic type parameter in order to instantiate.
Here it’s not working well in either the source file or the test runner. In the source file, you can see that it offers to run tests in a category, but not the tests from actual descendants. If you try to run or debug anything from this menu, it shows the fixture with a question-mark icon and marks any tests it manages to display as inconclusive. This is not surprising, since the test fixture may not be abstract, but does require a type parameter in order to be instantiated.
Here it looks and acts correctly:
I’ve reported this issue to JetBrains, but our testing structure either isn’t very common or it hasn’t made it to their core test cases, because neither 9 nor 10 handles them as well as the 8.x runner did.
Now that we’re also using TeamCity a lot more to not only execute tests but also to collect coverage results, we’ll capitulate and just change our patterns to whatever makes R#/TeamCity the happiest.
Once more to recap our ground rules for making tests:
TestFixture
only on leafs (classes with no descendants)Category
or Test
attributes anywhere in the hierarchy, but need to declare the class as abstract.When you make the change, you can see the improvement immediately.