“Change is the only constant in life.”
This quote is also true in the case of software application development. Nowadays software applications tend to evolve more frequently to accommodate the recent trends, technological updates, bug fixes, and improvements. Such applications require testing [r]evolution to maintain the quality of services provided by these applications.
In the web application domain, testers perform end-to-end testing of their applications by creating test scripts using Capture-Replay tools such as Selenium, TestComplete, QTP, and Watir. These test automation tools allow the testers to record the various test scenarios as a sequence of user actions (such as mouse clicks, keyboard entries, and navigation commands) performed on the web application and later replay (re-execute) these with the same or different data to test a web application.
One of the key benefits of test automation is reusability of test scripts across the product versions.
The reuse of existing test scripts for the updated version of the application helps in identifying and fixing regression bugs. It shortened the feedback loop by providing the confidence to introduce new changes without affecting the existing functionality.
Similar to other systems, web applications evolve over time. The evolution of web applications can introduce several different types of changes of web elements, for example, changes in visual representation due to new style sheets being applied, changes in visual labels for various fields, or distribution of fields on one page to multiple pages.
The test scripts of Capture-Replay tools are strongly coupled with the elements of web pages and are very sensitive to any change in the web elements of the web application. Even simple changes, such as slight modifications in a web page layout may break the existing test scripts, because the web elements that the scripts are referencing may not be valid in the new version. This overhead is considered a major obstacle for organizations to move towards automated web application testing.
Maintaining an automated test suite is important to avoid additional risks and difficulties for the testing team involved in the project. Unless the web application or the underlying software systems that run it never changes, it is required to ensure that automated tests continue to work well as the application evolves.
In case of evolving applications such as adding new features or fixing newly discovered bugs, will
The figure below shows the classification of the regression test suite into Reusable test scripts, Retestable test scripts, and Obsolete test scripts.
Figure 1: Classification of Regression Test Suite
Reusable test scripts refer to those where there is no change corresponding in the evolved version of the application, therefore these test scripts can be used directly.
Retestable test scripts are those test scripts that are affected by the changes in the evolved versions. Such test scripts could be fixed automatically by applying repair transformations or manually through testers.
Obsolete test scripts correspond to deleted web elements or web pages and hence are no longer valid for the evolved version.
Web Applications evolve over time
Fragile test case:Fragile tests are those that are easy to break. The test case was interrupted due to changes like date/time-sensitive assertions in the application.
Failed test case:A failing test is one that is well written and which has determined that the component being tested is broken. This usually happens when an assertion fails in the test case.
Smelly test case:Test smells are defined as bad programming practices in unit test code. For example, too many assertions or test dependency, etc.
Broken Test case:The
Bad Test:Bad tests are tests that don’t do their job properly. Either they’re testing more than they should, or they’re hanging on data that is inconsistent or subject to change based on external conditions.
Flaky test case:A flaky test is non-deterministic in nature that fails to produce the same result each time for the exact same application under the same test environment.
There are two types of approaches commonly used to fix or repair the broken test scripts of evolving web applications.
DOM-based Approach: This approach utilizes the DOM information of two versions of a web application and suggests potential fixes for the broken test commands to the tester. This technique typically collects the test case execution data for both the original and modified version of the web application.
It is based on differential testing and compares the behavior of the test case on two successive versions of the web application: the first version in which the test script runs successfully and the second version in which the script results in an error or failure. By analyzing the difference between the two executions, it suggests a set of repairs for the broken test scripts.
Visual Analysis based Approach: This approach leverages the visual information (such as screenshots) extracted through the execution of test cases, along with image processing and crawling techniques, to support automated repair of web test breakages.
Web applications are bound to evolve, and if that happens, test cases break. In this story, you have learned how to deal with test breakage to keep up with evolving web apps.
It’s all about the survival of the fittest…
You might also like: