xUnit: Managing Complex Test Parameters

xUnit supports many ways of parameterizing tests. However, none of the documentation I could find covers the use case of re-using and mixing parameter datasets across tests, which is particularly relevant when dealing with complex integration or automated UI tests.

In this article you will learn how this can be done with xUnit in Visual Studio. All source code is available.

Example

This is best explained with an example. Imagine you want to test a calculator. The calculator has two modes "scientific" and "normal" and exists in two models, the "T-800" and the mighty "T-900".

Some tests you want to exercise across all calculator modes and models, and some tests you want to exercise across modes and models plus an additional data set of numbers.

For example, to test addition you might want to add parameter sets like (-4, -6, -10) where each data set represents one addition calculation: -4 + -6 = -10.

So there exists 3 dimensions that you want to test across: modes, models, and numbers. Each of these dimensions is a set of parameters. Each test run can be thought of as a single point in the 3D space below:

Ideally, we want to define each dimension data set once, and then select what dimensions we want to use according to the test. Ideally, we also want to see correct information in the Visual Studio Test Explorer as to what parameters are going to be used.

Desired Solution

Imagine we have two tests. One is called CanAdd() that will test across all modes and models plus a list of number sets. The other is called CanSwitchOnOff() that will test across all modes and models. We want the Visual Studio test explorer to look like this:

We want the data sets to be defined only once.

Solution Implementation

The sample solution has one class for the calculator tests and a separate class for the parameters:

In the TestParameters class, we define each dimension's data set once. Our modes and models are simple strings, and the number data is a tuple of (int, int, int), where the sum of the first two integers should equal the third integer:

The system under test is a trivial calculator defined in the CalculatorTest class:

So we have our data sets and our system under test, how do we use these data sets to parameterize some tests? To do this, we define some "data mixers" that blend the desired data sets defined above. It is these mixers that will be referenced by the tests.

Defining the Data Mixers

The above mixers blend the desired data sets, sometimes you may want cartesian products (every combination) but sometimes you may not, but the data mixers give you complete control. Care must be taken to get the number and order of objects correct, since xUnit expects just an object array being passed to the test.

Using Mixers in Tests

The final part is to use the above mixers in the actual tests, as shown below:

Building the project and viewing the test explorer, shows the correct combination of test parameters under each test:

Thanks to discombobulator for the insight of using the yield keyword to achieve this design.