What if I told you The X-Men can make you a better software tester? Well they can’t but their supervillain, William Stryker, can.
This article is all about how to improve your unit tests using mutation testing. There are several mutation testing frameworks but the framework I’ll be talking about is called Stryker, aptly named after the supervillain from X-Men who wants to kill all of the mutants. I’ve been using Stryker for over two years now, in those two years I’ve implemented Stryker on three large enterprise projects and given several presentations on the framework.
I want to start this article with a warning, mutation testing is not for the faint of heart. At first, it’s tedious, time-consuming and often drags out the testing phase of development, often requiring developers to write even more unit tests. Why even bother with it? Because it is a foolproof way to ensure your unit tests are actually providing value.
Every developer has seen a unit test that looks great but actually doesn’t test anything or simply exists to provide code coverage. These are the tests that mutation testing exposes. It’s the cherry on top for unit tests, the pick of destiny, any rockstar tester isn’t a true rockstar unless they have mutation testing checking their unit tests. Now that you’ve been warned let’s get into the details.
The simple one sentence answer is mutation testing is testing for your unit tests. What does this mean? Mutation testing is done through a framework, like Stryker, that will mutate or change, specific parts of your code and check to see if that mutation causes a unit test to fail. If the mutant causes one of your existing unit tests to fail the mutant is considered “killed”. At the end of the mutation testing, a report is generated with a full list of killed and surviving mutants, similar to what you would see in a code coverage report using Jest or Jasmine.
Let’s say you have some code that is generating a URL for an application. Hopefully, there is a unit test checking that URL path and ensuring it gets generated. The mutation testing framework would come in and mutate your endpoint to a null or empty string. What happens to your unit test? If it fails that’s good, it means your test is smart enough to know what the URL should be and the mutant will be considered killed. If your unit test doesn’t fail that means your unit test isn’t quite as good as you thought. The framework will list the mutant as survived and you’ll need to go back to the drawing board to write a better unit test. Below is an example of what a string mutation would do:
String url = "deadpool.com"
String url = ""
When first starting to use mutation testing the process will be very slow. It can be difficult to kill certain mutants and having to rewrite unit tests is never fun. Over time the benefits are worth it though and just like anything in life the more you practice the faster the process will become.
Mutation testing still seems to be a rarity amongst developers but hopefully, this article has encouraged any developers reading to give it a shot. The setup may be slow and the first mutants may take a long time to kill but the more mutants killed the faster the process will become. Short of Wolverine offering unit testing tutoring, Stryker and mutation testing is as close as you can get to the X-Men improving your unit tests.
Create your free account to unlock your custom reading experience.