Are you ready for ReSharper 9? Not for testing, you aren’t.

We’ve been using ReSharper at Encodo since version 4. And we regularly use a ton of other software from JetBrains[1]—so we’re big fans.

How to Upgrade R#

As long-time users of ReSharper, we’ve become accustomed to the following pattern of adoption for new major versions:

EAP

1. Read about cool new features and improvements on the JetBrains blog
2. Check out the EAP builds page
3. Wait for star ratings to get higher than 2 out of 5
4. Install EAP of next major version
5. Run into issues/problems that make testing EAP more trouble than it’s worth
6. Re-install previous major version

RTM

1. Major version goes RTM
2. Install immediately; new features! Yay!
3. Experience teething problems in x.0 version
4. Go through hope/disappointment cycle for a couple of bug-fix versions (e.g. x.0.1, x.0.2)
5. Install first minor-version release immediately; stability! Yay!

This process can take anywhere from several weeks to a couple of months. The reason we do it almost every time is that the newest version of ReSharper almost always has a few killer features. For example, version 8 had initial TypeScript support. Version 9 carries with it a slew of support improvements for Gulp, TypeScript and other web technologies.

Unfortunately, if you need to continue to use the test-runner with C#, you’re in for a bumpy ride.

History of the Test Runner

Any new major version of ReSharper can be judged by its test runner. The test runner seems to be rewritten from the ground-up in every major version. Until the test runner has settled down, we can’t really use that version of ReSharper for C# development.

The 6.x and 7.x versions were terrible at the NUnit TestCase and Values attributes. They were so bad that we actually converted tests back from using those attributes. While 6.x had trouble reliably compiling and executing those tests, 7.x was better at noticing that something had changed without forcing the user to manually rebuild everything.

Unfortunately, this new awareness in 7.x came at a cost: it slowed editing in larger NUnit fixtures down to a crawl, using a tremendous amount of memory and sending VS into a 1.6GB+ memory-churn that made you want to tear your hair out.

8.x fixed all of this and, by 8.2.x was a model of stability and usefulness, getting the hell out of the way and reliably compiling, displaying and running tests.

The 9.x Test Runner

And then along came 9.x, with a whole slew of sexy new features that just had to be installed. I tried the new features and they were good. They were fast. I was looking forward to using the snazzy new editor to create our own formatting template. ReSharper seemed to be using less memory, felt snappier, it was lovely.

And then I launched the test runner.

And then I uninstalled 9.x and reinstalled 8.x.

And then I needed the latest version of DotMemory and was forced to reinstall 9.x. So I tried the test runner again, which inspired this post.[2]

So what’s not to love about the test runner? It’s faster and seems much more asynchronous. However, it gets quite confused about which tests to run, how to handle test cases and how to handle abstract unit-test base classes.

Just like 6.x, ReSharper 9.x can’t seem to keep track of which assemblies need to be built based on changes made to the code and which test(s) the user would like to run.

Abstract tests cannot be executedHere are the concrete test instances

To be fair, we have some abstract base classes in our unit fixtures. For example, we define all ORM query tests in multiple abstract test-fixtures and then create concrete descendants that run those tests for each of our supported databases. If I make a change to a common assembly and run the tests for PostgreSql, then I expect—at the very least—that the base assembly and the PostgreSql test assemblies will be rebuilt. 9.x isn’t so good at that yet, forcing you to “Rebuild All”—something that I’d no longer had to do with 8.2.x.

TestCases and the Unit Test Explorer

It’s the same with TestCases: whereas 8.x was able to reliably show changes and to make sure that the latest version was run, 9.x suffers from the same issue that 6.x and 7.x had: sometimes the test is shown as a single node without children and sometimes it’s shown with the wrong children. Running these tests results in a spinning cursor that never ends. You have to manually abort the test-run, rebuild all, reload the runner with the newly generated tests from the explorer and try again. This is a gigantic pain in the ass compared to 8.x, which just showed the right tests—if not in the runner, then at-least very reliably in the explorer.

Huge scrollbar in the unit-test explorerThe unit-test explorer likes to expand everything

And the explorer in 9.x! It’s a hyperactive, overly sensitive, eager-to-please puppy that reloads, refreshes, expands nodes and scrolls around—all seemingly with a mind of its own! Tests wink in and out of existence, groups expand seemingly at random, the scrollbar extends and extends and extends to accommodate all of the wonderful things that the unit-test explorer wants you to see—needs for you to see. Again, it’s possible that this is due to our abstract test fixtures, but this is new to 9.x. 8.2.x is perfectly capable of displaying our tests in a far less effusive and frankly hyperactive manner.

One last thing: output-formatting

Even the output formatting has changed in 9.x, expanding all CR/LF pairs from single-spacing to double-spacing. It’s not a deal-breaker, but it’s annoying: copying text is harder, reading stack traces is harder. How could no one have noticed this in testing?

Output formatting is double-spaced

Conclusion

The install/uninstall process is painless and supports jumping back and forth between versions quite well, so I’ll keep trying new versions of 9.x until the test runner is as good as the one in 8.2.x is. For now, I’m back on 8.2.3. Stay tuned.