In , the design of test cases is fundamental to ensuring the reliability and robustness of apps. Combinatorial test design techniques, such as k-way testing and pairwise testing, aim to test the interactions between different input variables. However, it becomes apparent that challenges and pitfalls can hinder their effectiveness in uncovering critical errors. software testing This short article focuses on specific issues that could be encountered during combinatorial test design, exploring the nuances that demand careful consideration. Let’s delve into instances of incorrect input values, the importance of robust oracles, the significance of often-overlooked variable combinations, and the crucial understanding of how variables interact. ncorrect Identification of Input Values I A k-way test set comprises all possible combinations of values for k variables. For instance, applying the Allpairs Algorithm results in a 2-way test set, encompassing all possible combinations of pairs of variable values. Consequently, a k-way fault refers to an error resulting from the interaction of values of k variables. Examining two systems, SYS1 and SYS2, revealed errors after their production releases. Both systems underwent k-way testing with k values of 2, 3, 4, and 5+. The table displays the outcomes of k-way testing for the two systems. Fault type SYS1 SYS2 2-way 30 29 3-way 4 12 4-way 7 1 > 4-way 7 3 Not Found 34 43 Sequential checks were conducted for each k-way set. Figure 2.1 indicates that errors arising from the interaction of two or fewer input variables were and . Errors from the interaction of three variables were and . For interactions involving four variables, errors were and . Errors from interactions with more than four variables were and . Notably, errors were not found in SYS1, and errors were not found in SYS2. 29 in SYS1 30 in SYS2 4*(30+4) in SYS2 12*(29+12) in SYS1 7*(30+4+7) in SYS2 1*(29+12+1) in SYS1 3*(29+12+1+3) in SYS1 7*(30+4+7+7) in SYS2 43 34 In this example, the most significant errors belonged to the category. Further investigations revealed that most of the undetected errors were 1-way faults, meaning a specific value of one variable led to the error independently of other variable values. Pairwise testing should have detected these errors, but for some reason, they remained undiscovered. These errors are primarily 1-way faults that went unnoticed because certain input values were either not selected or chosen incorrectly. Not Found Not Found lies in the fact that any errors made before the application of the Allpairs Algorithm will persist. This implies that if previous test design techniques were incorrectly applied or input values were incorrect, errors in the application will persist regardless of using the Allpairs Algorithm or Orthogonal Array Testing. The essence of the problem Weakness in Defining Expected Results: Uncertainty in Thest Outcomes As an example, let’s consider an application module with numerous options (something that looks like this form) and, consequently, a large number of input data combinations. In the module, let’s say that there are possible combinations. Here, the challenge is not in choosing incorrect input values, as all possible variable values can be exhaustively tested using the Allpairs algorithm. However, will this uncover all critical errors? Likely not, due to issues with expected results. After manipulating each combination of options in such kind of a software module, one would need to use the system for some time to verify that the options worked correctly and nothing else broke after applying the chosen options. In this case, there is no guarantee that there is no subtle, non-obvious problem easily missed. 2^12 * 3 After each test, one may need to thoroughly assess the core functions of the system. The point here is that serious defects are often not as apparent as one would like them to be, and accounting for everything in expected results becomes practically impossible. Inattention to the Most Probable Combinations Out of the combinations (as we suggested in the example above), there is likely one combination of module options that will occur much more frequently than all others - the setting. If 95% of people will never change the options from the default settings for this module, there should be good coverage of options. Testing all variations with a deviation from the default configuration in one option would result in a 2-digit number of tests. If testing all combinations of options with a possible deviation in two settings from the default, it would be around a hundred test cases. Testing with these test cases and all-pair test cases may yield better results than ignoring the existence of the default option. However, the Allpairs algorithm forces the tester to overlook the most probable and commonly used variable combinations. 2^12 * 3 default almost/near-default Unknown Variable Interactions The key factor for the success or failure of pairwise testing lies in understanding how the combination of input variables affects output data. Applying pairwise testing to two different tested applications may yield significantly different results. Some applications use fewer input data to produce output data, while others use more. As an example, consider the program, which has three input variables (X, Y, Z) and three possible output data (1, 2, 3). The arrows indicate which variables should interact to produce a specific result. To obtain a result of 1, variables Y and X must interact; to obtain a result of 2, variables Z and X must interact. For this application, applying pairwise testing would be a suitable choice, and positive results are likely. However, in cases where there are scenarios where output data result from the interaction of more than two input variables, there is a high likelihood that pairwise testing will fail to identify errors. Simply applying pairwise testing increases the risk of overlooking critical errors in the tested application. As a QA professional, understanding these nuances is important. While theoretical in some cases, these insights underscore the need for a holistic testing approach that goes beyond the surface, ensuring the reliability and robustness of your apps. Understanding the complexity of combinatorial test design allows QA professionals to effectively test the complicated business logic of apps and deliver high-quality software in to their users.

Read My Stories

Lead Software QA Engineer

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

Software QA: Solving Issues with Combinatorial Test Design

Too Long; Didn't Read

Constantine

About Author

TOPICS

Languages

THIS ARTICLE WAS FEATURED IN...

Software QA: Solving Issues with Combinatorial Test Design

Too Long; Didn't Read

Constantine

About Author

TOPICS

Languages

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES