Your browser may have trouble rendering this page. See supported browsers for more information.

This page shows the source for this entry, with WebCore formatting language tags and attributes highlighted.

Title

Improving NUnit integration with testing harnesses

Description

<img attachment="abstract_base_class.png" align="right">These days nobody who's anybody in the software-development world is writing software without tests. Just <i>writing</i> them doesn't help make the software better, though. You also need to be able to <i>execute</i> tests---reliably and quickly and repeatably. That said, you'll have to get yourself a test runner, which is a different tool from the compiler or the runtime. That is, just because your tests compile (satisfy all of the language rules) and <i>could</i> be executed doesn't mean that you're done writing them yet. <h>Testing framework requirements</h> Every testing framework has its own rules for how the test runner selects methods for execution as tests. The standard configuration options are: <ul> Which classes should be considered as test fixtures? Which methods are considered tests? Where do parameters for these methods come from? Is there startup/teardown code to execute for the test or fixture? </ul> Each testing framework will offer different ways of configuring your code so that the test runner can find and execute setup/test/teardown code. To write NUnit tests, you decorate classes, methods and parameters with C# attributes. The standard scenario is relatively easy to execute---run all methods with a <c>Test</c> attribute in a class with a <c>TestFixture</c> attribute on it. <h>Test-runner Requirements</h> <pullquote width="240px" align="right">There are legitimate questions for which even the best specification does not provide answers.</pullquote>When you consider multiple base classes and generic type arguments, each of which may also have NUnit attributes, things get a bit less clear. In that case, not only do you have to know what NUnit offers as possibilities but also whether the test runner that you're using <i>also understands and implements</i> the NUnit specification in the same way. Not only that, but there are legitimate questions for which even the best specification does not provide answers. At Encodo, we use Visual Studio 2015 with ReSharper 9.2 and we use the ReSharper test runner. We're still looking into using the built-in VS test runner---the continuous-testing integration in the editor is intriguing<fn>---but it's quite weak when compared to the ReSharper one. So, not only do we have to consider what the NUnit documentation says is possible, but we must also know what how the R# test runner interprets the NUnit attributes and what is supported. <h>Getting More Complicated</h> Where is there room for misunderstanding? A few examples, <ul> What if there's a <c>TestFixture</c> attribute on an abstract class? How about a <c>TestFixture</c> attribute on a class with generic parameters? Ok, how about a non-abstract class with <c>Tests</c> but no <c>TestFixture</c> attribute? And, finally, a non-abstract class with <c>Tests</c> but no <c>TestFixture</c> attribute, but there are non-abstract descendants that <i>do</i> have a <c>TestFixture</c> attribute? </ul> In our case, the answer to these questions depends on which version of R# you're using. Even though it feels like you configured everything correctly and it logically <i>should</i> work, the test runner sometimes disagrees. <ul> Sometimes it shows your tests as expected, but refuses to run them (Inconclusive FTW!) Or other times, it obstinately includes generic base classes that cannot be instantiated into the session, then complains that you didn't execute them. When you try to delete them, it brings them right back on the next build. When you try to run them---perhaps not noticing that it's those damned base classes---then it complains that it can't instantiate them. <i>Look of disapproval.</i> </ul> Throw the TeamCity test runner into the mix---which is ostensibly the same as that from R# but still subtly different---and you'll have even more fun. <h>Improving Integration with the R# Test Runner</h> At any rate, now that you know the general issue, I'd like to share how the ground rules we've come up with that avoid all of the issues described above. The text below comes from the <a href="https://secure.encodo.ch/jira/browse/QNO-5009">issue</a> I created for the impending release of Quino 2. <h level="3">Environment</h> <ul> Windows 8.1 Enterprise Visual Studio 2015 ReSharper 9.2 </ul> <h level="3">Expected behavior</h> Non-leaf-node base classes should never appear as nodes in test runners. A user should be able to run tests in descendants directly from a fixture or test in the base class. <h level="3">Observed behavior</h> Non-leaf-node base classes are shown in the R# test runner in both versions 9 and 10. A user must navigate to the descendant to run a test. The user can no longer run all descendants or a single descendant directly from the test. <h level="3">Analysis</h> Relatively recently, in order to better test a misbehaving test runner and accurately report issues to JetBrains, I standardized all tests to the same pattern: <ul> Do not use abstract anywhere (the base classes don't <i>technically</i> need it) Use the <c>TestFixture</c> attribute <i>only</i> on leaf nodes </ul> This worked just fine with ReSharper 8.x but causes strange behavior in both R# 9.x and 10.x. We discovered recently that not only did the test runner act strangely (something that they might fix), but also that the unit-testing integration in the files themselves behaved differently when the base class is abstract (something JetBrains is unlikely to fix). You can see that R# treats a non-abstract class with tests as a testable entity, even when it doesn't actually have a <c>TestFixture</c> attribute and even expects a generic type parameter in order to instantiate. Here it's not working well in either the source file or the test runner. In the source file, you can see that it offers to run tests in a category, but not the tests from actual descendants. If you try to run or debug anything from this menu, it shows the fixture with a question-mark icon and marks any tests it manages to display as inconclusive. This is not surprising, since the test fixture may not be abstract, but <i>does</i> require a type parameter in order to be instantiated. <img src="{att_link}no_abstract_base_class.png" href="{att_link}no_abstract_base_class.png" align="none" caption="Non-abstract base class" scale="75%"> Here it looks and acts correctly: <img src="{att_link}abstract_base_class.png" href="{att_link}abstract_base_class.png" align="none" caption="A test in an abstract test fixture" scale="75%"> I've reported this issue to JetBrains, but our testing structure either isn't very common or it hasn't made it to their core test cases, because neither 9 nor 10 handles them as well as the 8.x runner did. Now that we're also using TeamCity a lot more to not only execute tests but also to collect coverage results, we'll capitulate and just change our patterns to whatever makes R#/TeamCity the happiest. <h level="3">Solution</h> <ul> Make all testing base classes that include at least one {{Test}} or {{Category}} attribute {{abstract}}. Base classes that do not have any testing attributes do not need to be made abstract. </ul> Once more to recap our ground rules for making tests: <ul> Include <c>TestFixture</c> only on leafs (classes with no descendants) You can put <c>Category</c> or <c>Test</c> attributes anywhere in the hierarchy, but need to declare the class as abstract. Base classes that have no testing attributes do not need to be abstract If you feel you need to execute tests in both a base class and one of its descendants, then you're probably doing something wrong. Make two descendants of the base class instead. </ul> When you make the change, you can see the improvement immediately. <img src="{att_link}after_making_base_class_abstract.png" href="{att_link}after_making_base_class_abstract.png" align="none" caption="After making the base class abstract" scale="66%"> <hr> <ft>ReSharper 10.0 also offers continuous integration, but our experiments with the EAP builds and the first RTM build left us underwhelmed and we downgraded to 9.2 until JetBrains manages to release a stable 10.x.</ft>