<h>Introduction</h> Testing is any form of validation that verifies a product. That includes not only structured validation using checklists, test plans, etc. but also informal testing, as when engineers click their way through a UI, emit values in debugging output to a console, or perform operations on hardware. <i>Automated testing</i> is common for software, as regression-style tests that execute both locally and in CI. This includes unit, integration, and end-to-end tests. The following discussion focuses primarily on _software-testing_ but hopefully contains some insights and information relevant to other engineering disciplines (e.g., embedded and hardware developers). <h>The testing mindset</h> Testing is primarily a mindset. Thinking about what you're building in the terms outlined above can help you to determine how and what you're actually going to build. It will help you focus, <ul> going from "this would be a nice feature" 🤩 to "how would I test it?" 🤨 to "who would actually use it?" 🙄 to, perhaps, "it would be neat, but no-one needs it. It's not a requirement." ✋ or, "the use case is clear and here is how I would test it." 👌 </ul> You should think of writing tests not as something you _have_ to do, but rather as something you _want_ to do. <ul> How else do you prove that what you've made actually works? What does <i>"it works"</i> mean? Which <i>use cases</i> are covered? How do you answer these questions without tests? What do we mean by <i>writing</i> tests? </ul> Let's define some of this jargon—use cases? "it works", etc.—before we continue. <h>Why do we test?</h> It's a bit of a provocative question, perhaps, but it makes sense to ask about anything into which you're going to invest time and money. So, let's start a bit further back. ❓ What would we like to do? <bq>We would like to build a <i>product</i> of high <i>quality</i></bq> ❓ What's a product? <bq>A <i>product</i> is an implementation of a set of <i>requirements</i>.</bq> ❓ Then what's a requirement? <bq>A <i>requirement</i> is a collection of <i>use cases</i>.</bq> ❓ OK, fine. What's a use case? <bq>A <i>use case</i> comprises a set of initial conditions, an action, a set of inputs, and an expected output.</bq> ❓ What is quality? <bq>A product that satisfies its <i>requirements</i> is of higher <i>quality</i> than one that does not.</bq> ❓ How can I know that my product has the desired quality? <bq>We <i>test</i> <i>use cases</i> for a version of a <i>product</i> to determine <i>quality</i>.</bq> ❓ How can I know when my product has enough tests? <bq>When all of the <i>use cases</i> are <i>covered</i>.</bq> ❓ What if I change the product after I've tested it? <bq>Then you have to test all of the use cases again.</bq> ❗ What the heck? That's boring! I don't have time for that! <bq>It's called regression-testing. There's no way around it.</bq> ❓ What if I know that I've only changed a tiny thing? <bq>You might be able to get away with it. But that's where 🕷 <i>bugs</i> come from.</bq> ❗ I can't afford to test everything manually every time I make a change! <bq> That's why you automate as many tests as you can.</bq> ❗ Running the tests ties up my local machine! I can't work. <bq>Run tests in another environment (e.g., in the cloud)</bq> <h level="3">Conclusions</h> <ul> A <i>product of quality</i> includes <i>tests</i>. A <i>product</i> is considered <i>untested</i> if it has changed since it was last <i>tested</i>. <i>Regression-testing</i> is unavoidable. Automated tests improve <i>efficiency</i> and <i>reliability</i> Using a separate environment improves <i>robustness</i> </ul> <h>Introduction to methodologies</h> We've established both that testing is a mindset and that it is necessary to building high-quality products. We should keep in mind that the <i>goal</i> is to have a well-tested product with as many of these tests as possible being automated. The question is: how close to the goal state do you stay <i>during development?</i> <h level="3">Developer feedback loop</h> In other words, what does the development-feedback loop look like? The goal of the development-feedback loop is to shorten the time between a change and its verification. In practice, this often manifests as "knowing as soon as possible when you've broken something." The longer it takes from change to verification, the more likely it is that multiple changes will be verified at once. Root-cause analysis becomes more difficult. That's why manual tests are undesirable: they are far less likely to be run/applied in a timely manner, increasing the number of changes that have occurred since the last time tests were run. So, the longer you wait to define tests, the longer your product remains untested. The longer you wait to <i>automate</i> tests, the longer you must do manual testing to verify behavior. With that idea in mind, let's consider the spectrum of methodologies. At one end, there's <a href="https://en.wikipedia.org/wiki/Test-driven">development">TDD</a>, where you write the tests <i>first</i>, letting them fail and <i>then</i> writing the implementation. At the other, there's writing all of your acceptance tests once you've finished the product. <h level="4">Test-driven Design</h> Always writing the tests first is just one extreme, and one that scares a lot of people away from automated testing. As with any dogma, strict adherence is unlikely to be efficient. Sometimes, you'll need to try out an implementation to see if it's even feasible or want to play with an API to see how it feels before you write a ton of tests for it. You don't want to go too long without testing that you haven't broken something, but you also don't want to write tests for code that you're going to throw away in an hour anyway. Tests are only one part of the array of techniques a developer can use to verify a product. As discussed in more detail below, a strong type system, linting, and static-code analysis of all kinds help verify a product. We should always be aware of which parts are necessary <i>during which phases</i>. If certain tools take longer to verify code, consider whether they need to be executed all the time, or perhaps just when pushing to a remote, or before merging into the master branch. <h level="4">Acceptance-tests at the end</h> If you wait until you've finished the product to write all of your tests, you will still have a well-tested product, but you will not have benefited from testing during development. Being able to test as you go improves your efficiency tremendously, as you're not constantly fighting with things that are mysteriously breaking. Instead, you're usually able to pin the blame on <i>the most recent change you've made.</i> A product of nontrivial complexity can be written more reliably and quickly if there are tests. It also becomes possible for one team member to write the tests while another provides the implementation that satisfies it. <h level="4">A balanced approach</h> The spectrum in between is where most developers live, writing tests as they go, but not always <i>before</i> they've implemented something. It's understandable that there will always be certain tests that are difficult, if not impossible to automate. However, the document that follows will provide some tools for extracting the testable bits from the untestable ones to increase <i>coverage</i>. Anything that can be tested automatically can be executed by all team members all the time, as well by pipelines in the cloud. <h>You're already testing!</h> You're almost certainly already testing. You might be clicking through the UI or emitting statements in a command-line application, but you're verifying your code <i>somehow</i>. I mean ... you are, right? RIGHT? I'm kidding. Of course you're not just writing code, building it, and committing it. You're validating it somehow. That's testing. <h level="3">A list of validations</h> If you're really good, you might even keep a list of these validations. Once you have a list, then, <ol> You don't have to worry about forgetting to do them in the future Even someone with no knowledge of the system can perform validation </ol> This is fine, but it's still a manual process. A manual process carries with it the following drawbacks: <ol> It gets quite time-consuming, especially as the list of validations grows You're highly unlikely to perform the validations often enough<ul> It's much easier to fix a mistake if you learn about it relatively soon after you made it </ul> You're also unlikely to add <i>all</i> of the validations you need<ul> Generally, you won't validate smaller "facts" and will focus on high-level stuff</ul> You're much more likely to make mistakes in manual testing A manual validation process can't be run as part of CI or CD </ol> <h level="3">Automating the list</h> Automated testing means that you <i>codify</i> those validations. <info>😒 Great! I have tests! How the heck do I <i>codify</i> them?</info> Don't panic. Almost any code can be tested. In fact, if you can't get at it with a test, then you might have found an architectural problem. See? Automating tests will even help you write better code! <info>🤨 How do I get started?</info> Just start somewhere. It doesn't matter where. Don't worry about coverage. Just get the feeling for writing a proof about a facet of your code. Any bit of logic can—and should—be tested. What if you still don't know where to begin? Ask someone for help! Don't be shy. It's in everyone's best interest for a project to have good tests. You want everyone's code to have tests so you know <i>right away</i> when you've broken something in a completely unrelated area. This is a good thing! <h>Goals</h> <info>🤸‍♀️ Developers should be excited to use tests to prove that their code works.</info> <h level="3">Tests should be quick and easy (maybe even fun) to write</h> A project should provide support for mocking devices and external APIs, or for using test-specific datasets. <h level="3">Tests should be reasonably fast</h> A reasonably fast test suite will tend to be run more often. We would like a developer to notice a broken test right after the change that broke it, preferably even before pushing it. <h level="3">Avoid debugging tests in CI</h> Tests a developer runs locally should almost always work in CI. Failing tests in CI should also fail locally. <h>Guidelines</h> <info>🤨 Don't be pedantic.</info> For example, <ul> Don't get obsessed with automating <i>everything</i>.<ul> Get the low-hanging fruit first, and leave the rest to manual testing. See where you stand. If you haven't automated enough, iterate until done. 🔄</ul> Don't forbid mocking in integration tests and don't force mocking in unit tests.<ul> In fact, stop worrying about whether it's a unit or an integration and just <i>write useful tests</i> that <i>prove useful things</i> about your code.</ul> <a href="https://stackoverflow.blog/2022/11/03/multiple-assertions-per-test-are-fine/">Stop requiring only one assertion per unit test: Multiple assertions are fine</a> </ul> <h level="3">Figure out where you stand</h> The following questions should help you evaluate for yourself where you are on your automated-testing journey. <ul> How much automated testing have you done? Do you write automated tests now? Do you feel confident that you can verify your work with automated tests? Do you understand the limitations? Do you understand how system architecture can affect testability? </ul> <h level="3">Tests should be useful</h> We never want anyone in a team to get the impression that we're writing tests just to write tests. We write tests because they help us write better code and because it feels good to be able to prove that something that was working continues to work. You should feel more efficient and productive and feel like you're producing higher-quality code. <ul> Tests should confirm use cases Tests should prove something about your code that you think is worth proving. Tests should confirm behavior that either is how the code <i>currently</i> works or how it <i>should</i> work. Tests should help you write better code from the get-go. Every bug that you need to fix is de-facto a use case that needs a test. </ul> <h level="3">Code Coverage & Reviews</h> How do you know when there are "enough" automated tests? Don't get distracted by trying to achieve a specific coverage percentage. The most important thing is that the major use cases are covered. If software is stable and there is "only" 40% test-coverage, then maybe there is a lot of code that rarely or never gets used? In that case, you might want to think about removing code that you don't need rather than to waste time writing tests for code that never runs. New code, though, should always have automated tests. A <b>code reviewer</b> should verify that new functionality is being tested. <h>Types of tests</h> <dl dt_class="field"> Unit <div>Cover a single unit, mocking away other dependencies where needed. Useful for verifying simple logic like calculated properties or verifying the results of service methods with given inputs. </div> Integration <div>Cover multiple units, possibly mocking unwanted dependencies Useful for verifying behavior of units in composition, as they will be used in the end product. The goal is to cover as much as possible without resorting to more costly end-to-end tests </div> End-to-End <div>Also called <i>UI Tests</i>, these tests verify the entire stack for actual customer use cases Very useful, but generally require more maintenance as they tend to be more fragile. Essential for verifying UI behavior not reflected in a programmatic model. Can work with snapshots (e.g. error label is in red) </div> </dl> <h>Approach</h> The article <a href="https://kentcdodds.com/blog/write-tests">Write tests. Not too many. Mostly integration.</a> describes a pragmatic approach quite well. Instead of the classic "testing pyramid", it suggests a "testing trophy". <img src="{att_link}testing_trophy.jpg" href="{att_link}testing_trophy.jpg" align="none" scale="50%"> This style of development has the following aims: <ol> Verify as much as possible <i>statically</i>, with linting and analyzers Make <i>integration tests</i> cheaper because they prove more about your system than <i>unit tests</i> Prove as much as possible outside of <i>end-to-end tests</i> because they're expensive and brittle </ol> <h>Analysis</h> <info>Remember that everything you use has to work both locally and in CI.</info> <h level="3">Static-checking</h> A project should include analyzers and techniques so that the compiler helps make many tests unnecessary. For example, if you know that a parameter or result can never be <c>null</c>, then you can avoid a whole slew of tests. Developers should only spend time writing tests that verify semantic aspects that can't be proven by the compiler. <h level="4">Null-reference analysis in .NET</h> The .NET world provides many, many analyzers and tools to verify code quality. One of the most important things a project can do is to improve null-checking. The best way to do this is to upgrade to C# 8 or higher and enable <a href="https://learn.microsoft.com/en-us/dotnet/csharp/nullable-references">null-reference analysis</a>. The <a href="https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/configure-language-version">default language for .NET Framework is going to stay C# 7.3</a>, but you can <a href="https://www.infoq.com/articles/CSharp-8-Framework/">enable null-reference analysis for .NET Framework</a> quite easily. Another option is to use the <a href="https://www.nuget.org/packages/JetBrains.Annotations/">JetBrains Annotations NuGet package</a>, which provides attributes to indicate whether parameters or results are nullable. The preferred way, though, is to use the by-now standard nullability-checking available in .NET. Doing neither is not a good option, as it will be very difficult to avoid null-reference exceptions. <h level="3">Unit-testing</h> Unit tests are very useful for validating <i>requirements</i> and <i>invariants</i> about your code. These are the easiest tests to write and will generally be the first ones that you will write. A requirement or an invariant may be specified in the story itself, but it can be .anything that you know about the code that's important. It's up to the developer and the reviewer(s) to determine which tests are necessary. It gets easier with experience—and it doesn't take long to get enough experience so that it's no longer so intimidating. <h level="4">Unit-testing example</h> Just as a quick example in .NET, consider the following code, <code>public bool IsDiagnosticModeRunning { get => _isDiagnosticModeRunning; set { _isDiagnosticModeRunning = value; _statusManager.InstrumentState = value ? InstrumentState.DiagnosticMode : InstrumentState.Ready; } }</code> Here we see a relatively simple property with a getter and a setter. However, we also see that there is an invariant in the implementation: that the <c>_statusManager.InstrumentState</c> is synced with it. Using many of the <a href="#tools-and-techniques">techniques described below</a>, we could write the following test: <code>[DataRow(true, InstrumentState.DiagnosticMode)] [DataRow(false, InstrumentState.Ready)] [TestMethod] public void TestIsDiagnosticModeRunning(bool running, InstrumentState expectedInstrumentState) { var locator = CreateLocator(); var instrumentControlService = locator.GetInstance<iinstrumentcontrolservice>(); var statusManager = locator.GetInstance<istatusmanager>(); Assert.AreNotEqual(expectedInstrumentState, statusManager.InstrumentState); instrumentControlService.IsDiagnosticModeRunning = running; Assert.AreEqual(expectedInstrumentState, statusManager.InstrumentState); }</code> Here, we're using MSTest to create a parameterized test that, <ul> creates the IOC gets the two relevant services from it Verifies that the state is not already set to the expected state (in which case the test would succeed even if the tested code doesn't do anything) Sets the property to a given value Verifies that the state is correct for that value </ul> We now have code that validates two <i>facts</i> about the system. Should something change where these facts are no longer true, the tests will fail, giving the developer a chance to analyze the situation. <ul> Was the change inadvertent or deliberate? Are the facts still correct? Does the test need to be updated? </ul> If you're addressing a bug-fix, though, you might be able to <i>prove</i> that you've fixed the bug with a unit test, but it's also likely that you'll have to write an integration test instead. <h level="3">Integration-testing</h> Unit tests have their place, but they are far too emphasized in the testing pyramid. The testing pyramid comes from a time when writing integration tests was much more difficult than it (theoretically) is today. The "theoretically" above means that the ability to write integration tests as efficiently as unit tests is contingent on a project offering proper tools and support. One common complaint about integration tests vis à vis unit tests is that they run more slowly. Another is that they take longer to develop. Ideally, a project provides support to counteract both of these tendencies. To this end, then, a project should offer base and support classes that make common integration tests easy to set up and quick to execute: <ul> Interacting with a database Setting up a known database schema Getting to a clean dataset <a href="">Mocking</a> the database Mocking other external dependencies in a project (e.g. loading configuration from an endpoint, sending emails, sending modifications to endpoints) </ul> There are many different ways to solve this problem, each with tradeoffs. For example, a project can load dependencies in Docker containers, either created and started manually (see <a href="https://josef.codes/testing-your-asp-net-core-application-using-a-real-database/">Testing your ASP.NET Core application - using a real database</a>) or even dynamically with a tool like the <a href="https://github.com/testcontainers/testcontainers-dotnet">Testcontainers NuGet package</a>. <h level="3">Comparing Unit and Integration tests</h> A drawback to unit tests is that, while they can test an individual component well, it's really the big picture that we want to test. We want to test scenarios that correspond to actual use cases rather than covering theoretical call stacks. It's not that the second part <i>isn't</i> important, but that it's not <i>as</i> important. Given limited time and resources, we would prefer to have integration tests that also cover a lot of the same code paths that we would have covered with unit tests, rather than to have unit tests, but few to no integration tests. This, however, leads directly to... The advantage of a unit test over an integration test is that when it fails, it's obvious which code failed. An integration test, by its very nature, involves multiple components. When it fails, it might not be obvious which sub-component caused the error. If you find that you have integration tests failing and it takes a while to figure out what went wrong, then that's a sign that you should bolster your test suite with more unit tests. Once an integration test fails <i>and</i> one or more unit tests fail, then you have the best of both worlds: you've been made aware that you've broken a use case (integration test), but you also know which precise behavior is no longer working as before (unit test). <h>Tools and Techniques</h> <h level="3">Tests are Code</h> Testing code is just as important as product code. Use all of the same techniques to improve code quality in testing code as your would in product code. Clean coding, good variable names, avoid copy/paste coding---all of it applies just as much to tests. There are two main differences: <ul> You don't need to document tests You don't have to write tests for tests. :-) </ul> <h level="3">Writing testable code</h> This is a big, big topic, of course. There are a few guidelines that make it easier to write tests—or to avoid having to write tests at all. As noted above, code that can be validated by the compiler (static analysis) doesn't need tests. E.g. you don't have to write a test for how your code behaves when passed a <c>null</c> parameter if you just <i>forbid it</i>. Likewise, you don't have to re-verify that types work as they should in statically typed languages. We can trust the compiler. Here are a handful of tips. <ul> Prefer composition to inheritance A functional programming style is very testable An IOC Container is very helpful Avoid nullable properties, results, and parameters Avoid mutable data Interfaces are much easier to fake or mock; use those wherever you can Generally, the <a href="https://en.wikipedia.org/wiki/SOLID" source="Wikpedia">SOLID</a> principles are a decent guide </ul> See the following articles for more ideas. <ul> <a href="https://github.com/mvonballmo/CSharpHandbook/blob/master/4_design.md">C# Handbook Chapter 4: Design (2017)</a> <a href="{app}/view_article.php?id=2996">Questions to consider when designing APIs: Part I (2014)</a> <a href="{app}/view_article.php?id=2997">Questions to consider when designing APIs: Part II (2014)</a> <a href="{app}/view_article.php?id=3487">Why use an IOC? (hint: testing) (2019)</a> </ul> <h level="3">Parameterized Tests</h> Investigate your testing library to learn how to write multiple tests without having to write a lot of code. In the MSTests framework, you can use <c>DataRow</c> to parameterize a test. In NUnit, <c>TestCase</c> does the same thing, and <c>Value</c> allows you to provide parameter values for a list of tests that are the Cartesian product of all values. <h level="3">Mocking/Faking</h> Use mocks or fakes to exclude a subsystem from a test. What would you want to exclude? While you will want to make some tests that include database access or REST API calls, there are a lot of tests where you're proving a fact that doesn't depend on these results. <h level="4">Focus on what you're testing</h> For example, suppose a component reads its configuration from the database by default. A test of that component may simply want to see how it reacts with a given input to a given method. Where the configuration came from is irrelevant to that particular test. In that case, you could mock away the component that loads the configuration from the database and instead use a fake object that just provides some standard values. <h level="4">Test error conditions</h> Another possibility is to fake an external service to see how your code reacts when the service returns an error or an ambiguous response. Without mocks, how would you test how your code reacts when a REST endpoint returns 503 or 404? Without a mock, how would you force the purely external endpoint to give a certain code? You really can't. With a mock, though, you can replace the service and return a 404 response for a specific test. This is quite a powerful technique. <h level="4">How to fake?</h> As noted above, it's much, much easier to use fake objects if you've consistently used interfaces. You can just create your own implementation of the interface whose standard implementation you want to replace, give it a fake implementation (e.g. returning <c>false</c> and empty string and <c>null</c> for methods and properties) and then use that class as the implementation. <h level="4">Faking/mocking libraries</h> If you have interfaces that perform a single task (single-responsibility principle), then it doesn't take too much effort to write the fake object by hand. However, it's much easier to use a library to create fake objects—and there are other benefits as well, like tracking which methods were called with which parameters. You can assert on this data collected by the fake object. For .NET, a great library for faking objects is <a href="https://fakeiteasy.github.io/">FakeItEasy</a>. With a fake object, you can indicate which values to return for a given set of parameters without too much effort. Similarly, you can use the same API to query how often these methods have been called. This allows you to verify, for example, that a call to a REST service <i>would have been made</i>. This is a powerful way of proving facts about your code without having to actually interact with external services. <h level="4">An example</h> The following code configures a fake object for <c>ITestUnitConfigurationService</c> that returns default data for all properties, except for <c>Configuration</c> and <c>GetTestUnitParameterValues()</c>, which are configured to return specific data. <code>private static ITestUnitConfigurationService CreateFakeTestUnitConfigurationService() { var result = A.Fake<itestunitconfigurationservice>(); var testUnitParameters = CreateTestUnitParameters(); var testUnitConfiguration = new TestUnitConfiguration(testUnitParameters); A.CallTo(() => result.Configuration).Returns(testUnitConfiguration); var testUnitParameterValues = CreateTestUnitParameterValues(); A.CallTo(() => result.GetTestUnitParameterValues()).Returns(testUnitParameterValues); return result; }</code> In the test, we could get this fake object back out of the IOC (for example) and then verify that certain methods have been called the expected number of times. <code>var testUnitConfigurationService = locator.GetInstance<itestunitconfigurationservice>(); A.CallTo(() => testUnitConfigurationService.Configuration).MustHaveHappenedOnceExactly(); A.CallTo(() => testUnitConfigurationService.GetTestUnitParameterValues()).MustHaveHappenedOnceExactly();</code> <h level="3">Snapshot-testing</h> You can avoid writing a ton of assertions and a ton of tests with snapshot testing. For example, imagine you have a test that generates a particular view model. You want to verify 30 different parts of this complex model. You <i>could</i> navigate the data structure, asserting the 30 values individually. That would be pretty tedious, though, and lead to fragile and hard-to-maintain testing code. Instead, you could emit that structure as text and save it as a <i>snapshot</i> in the repository. If a future code change leads to a different snapshot, the test fails and the developer that caused the failure would have to approve the new snapshot (if it's an expected or innocuous change) or fix the code (if it was inadvertent and wrong). The upside is that large swaths of assertions are reduced to a simple snapshot assertion. The downside is that the test might break more often for spurious reasons. Generally, you can avoid these spurious reasons by being judicious about how your format the snapshot, <ul> Avoid timestamps or data that changes over time Avoid using output methods that are too likely to change over time </ul> See the documentation for the <a href="https://swisslife-oss.github.io/snapshooter/">Snapshooter NuGet package</a>. <h level="3">End-to-end Testing</h> There have been many solutions to the problem of automated testing of web UIs over the years. The one many know is <a href="https://www.lambdatest.com/selenium">Selenium</a>, but tools like <a href="https://www.cypress.io/">Cypress</a>, <a href="https://testcafe.io/">TestCafe</a>, <a href="https://pptr.dev/">Puppeteer</a> and <a href="https://playwright.dev/">Playwright</a> have largely replaced it. The <a href="https://webdriver.io/">WebdriverIO</a> library Before choosing a tool, you'll want to consider what your requirements are: <ul> Tests should run quickly Headless/command-line support for integrating into CI builds A GUI for running tests is a plus Traceability of tests Snapshot-testing Debugging, including rewinding through the UI events </ul> The current front-runner for end-to-end testing is <a href="https://playwright.dev/dotnet/">Playwright</a>, an open-source cross-browser, cross-platform, cross-language testing framework. <ul> Video: <a href="https://youtu.be/jF0yA-JLQW0">What's new in Playwright 1.32</a> shows the new _UI Mode_ in action (see the <a href="https://github.com/microsoft/playwright/releases/tag/v1.32.0">release notes</a>; screenshot below) !<a href="/.attachments/image-83c94c5b-b517-4e8b-907e-5791bb6e4cc2.png =400x">image.png</a> Video: <a href="https://youtu.be/sRjN-CU_Lg4">"Playwright can do this?" — Microsoft meetup March 2023</a> (see masking for visual regression at 00:18:00) <a href="https://github.com/microsoft/playwright">GitHub</a> <a href="https://learn.microsoft.com/en-us/microsoft-edge/playwright/">Example</a> </ul> <h level="3">Planner / Executor Pattern</h> This pattern is particularly useful when you have a bunch of <i>steps</i> to execute. Instead of executing the steps as you go, you build a <i>plan</i> that describes how those steps would be executed and return that as the result of the <i>planner</i> phase. You can test this plan very easily without worrying about how to mock away the <i>mutating</i> part of the code. For example, suppose you want to sync an online data source with a local configuration. The classic way would be to do something like the following: <code> var items = GetItemsFromServer(); foreach (var item in items) { var itemData = GetItemDataFromServer(item); if (string.IsNullOrEmpty(itemData.Text)) { SetStandardText(item, itemData); SaveItemToServer(item); } } </code> With so little logic, there's really no way to question this setup, is there? But think about what happens if there are more decisions to make, more data to retrieve, more data to update on the server. As this logic increases in complexity, the mutating code becomes ever more deeply embedded in read-only logic. That read-only logic ends up being the lion's share of the code that you want to test, but you have to step very lightly to avoid making changes on the server. You can, of course, mock away services, to make sure that nothing is communicated back to the server, but there is another way. What if you were to consider the set of operations as phases? <ol> A <i>planning</i> phase where the program gathers all of the information that it needs to determine which commands to execute in order to "repair" the situation. A much shorter and simpler <i>execution</i> phase where the program loops over the plan and applies it. </ol> This approach has several advantages: <ul> There are fewer questions about how to handle exceptions that occur while applying the plan. You don't have to worry about what happens when a mutation occurs deep within the planning logic. It's easier to test the meat of the logic because the output is a plan that you can <i>snapshot</i> or otherwise verify. You have the user-friendly option to present the user with a detailed plan of what will happen before applying any changes. You can even store the plan to execute later, e.g., after it has been audited by a separate team. </ul> Once again, we have a pattern that not only makes testing easier, but it makes the entire architecture more robust, opening up possibilities that you wouldn't have with the straightforward pattern (which would be harder to test). To finish up this section, let's take a quick look what that could look like in pseudocode. <code> var items = GetItemsFromServer(); var commands = new Commands(); foreach (var item in items) { var itemData = GetItemDataFromServer(item); if (string.IsNullOrEmpty(itemData.Text)) { var command = CreateCommand( "Set standard text for {item}", () => { SetStandardText(item, itemData); SaveItemToServer(item); } ) } } // Present commands to the user; store the commands for later, or execute them... // This is where tests would verify the commands generated from a given set of // item data. foreach (var command in commands) { try { command.Apply(); } catch { // Log error and continue? } } </code> Instead of executing the command immediately, we store what we would want to do with a closure and a description. We can do whatever we want with those commands; executing this is one option, but you can see how useful it would also be for verifying that the logic is correct in tests.