CI-friendly options in the world of automation testing tools

Few would disagree that CI support is highly important for UI testing automation solutions, as it adds one more line of defense against bugs and errors. Let’s see what the market of QA automation tools has to offer in terms of CI support, and how many of these solutions are good at dealing with visual regression testing challenges for web UIs.

Does anyone need to integrate visual testing with CI?

Here’s what a typical process looks like: one job creates a build, while another launches unit tests on it, allowing early spotting of most easy-to-find problems. If the build passes this stage successfully, testers can step in with their usual routine. They’ll take a quick look at the UI and, if everything is fine, proceed with functional testing.

Now, this process seems to have a lot of room for improvement, doesn’t it?

In particular, it would be super awesome to automate this quick UI sanity check, as well as the overall visual testing routine. Let another CI job run automated visual tests against a fresh build.

Sounds fantastic, but this task requires a reliable QA automation tool with CI support. So what are the tools that we get to choose from?

Existing UI testing automation solutions for running Selenium on CI

Let’s start with arguably the most well-known combo, Selenium + Jenkins.

You’ve probably noticed how many homemade Selenium-for-Jenkins solutions are out there, used by various development teams all over the globe. Basically, each of these solutions aims to offer an automation testing tool that’s optimal for a specific environment. However, there are numerous complexities that are common for most solutions of this kind:

• You will need to set up another testing framework (like jUnit) and use it as a base.
• Selenium Grid is often necessary because running the whole load of Selenium tests on one computer is problematic.
• Most probably, at least several of these instances will require separate fine-tuning of access rights (e.g. when enabling Jenkins starting browsers).
• You have to spend time making sure that all ports that Jenkins and Selenium are using by default are free on each node of your network.
• Finally, you’ll need to run this system in ‘testing’ mode a couple of times to detect any possible issues; then throw in more code to solve them, create additional batch files etc before starting full-scale UI testing automation.

Now, doesn’t this all look like like a huge pile of work? Luckily, the Jenkins team has come up with their own solution, which makes things a bit simpler. They have updated it in the summer of 2016, and it seems that many users have come to download it. The situation is pretty much the same for TeamCity, Travis and Bamboo. Still, a problem that remains is: Selenium itself does not provide a unified solution for CI integration.

UI automation testing with TestComplete plus Jenkins

Another big automation testing tool TestComplete has its own solution for integration with Jenkins. Yet, a unified integration plugin still remains a dream for its users who have to invent their own means to integrate their UI automation testing processes with CI platforms.

This may look tiresome enough unless a team of developers and network integrators is assigned to that task. And it is just the beginning.

Once you’re done setting up your CI environment, it’s time to start thinking about the maintenance of your tests. As your project progresses, some of your tests will inevitably get updated or replaced. Aside from that, even a small change to your project may bring upon the need to reorganize the whole network.

A less technical person (such as a middle QA engineer with mostly manual testing experience or a business person) will face a number of complexities while doing this job. In fact, these complexities can be so huge that, ultimately, they will cause a delay in the process of UI testing automation. No wonder that so many teams today still prefer hiring a couple of additional manual junior QAs to avoid investing too much at the early development stage.

Selenium alternatives and new emerging tools for testing automation: how they are address this problem

It’s no secret that Selenium has myriads of fans all over the world. Many of them coordinate their efforts in developing some local improvements, and the latter get shared with the public.

Eventually, Selenium and other prominent automation testing tools such as Cucumber and TestComplete get enough 3rd party solutions for their CI integration which would meet most of their users’ needs. So, is there a problem? Yes — the UI testing automation niche needs a simpler and more efficient solution.

Visual regression testing automation is the most obvious niche were low-level frameworks fail to deliver. The fact is, traditional platforms such as Selenium and Cucumber cannot perform this kind of web UI verification. They can perform functional testing and check for certain fields to be present, however they cannot detect a changed font, a wrong text or even a broken image. But these bugs are no less important!

So what are the alternatives to Selenium when it comes to running automated tests in continuous integration environment?

PhantomCSS

One of the most popular options is PhantomCSS. Unlike Selenium, this automation testing tool is actually capable of CSS testing. What’s more, it can compare web pages in a screenshot-by-screenshot format. Whenever a test step fails, Phantom displays a screenshot to a user.

As for the downsides, PhantomCSS is a cumbersome solution. Its installation implies downloading and setting up several separate modules, and you will also need to fine-tune the whole framework once you’re through with its components. Basically, there’s a lot to be done before you can get down to creating your tests.

Besides, Phantom uses its own built-in browser emulator and does not run in real Chrome or Firefox. This means that it’ll fail to trace browser-specific CSS issues.

Yet another common problem with Phantom is the learning curve. Similarly to Selenium and TestComplete, PhantomJS requires an experienced developer to deploy everything smoothly and to maintain tests. But this means that UI testing automation is actually performed by programmers, while non-coders are left outside the game. Wouldn’t it be more efficient to enable testers to create and manage all tests?

Finally, what about CI integration? As a matter of fact, Phantom itself does not provide any special integration tools with any of the popular CI clients. It can be installed separately on a slave/build instance of Jenkins or TeamCity so that its tests can run there. Another client, Travis, has created its own plugin for Phantom, but it has no multi-vendor CI integration solution either.

Screenster

Screenster, a new CSS automation testing tool (officially came out of beta on June 1, 2016), is currently the only project on the market that can boast full support for all major CI tools including Jenkins, TeamCity, Travis and Bamboo. Thanks to the simplicity of Screenster’s functional set, its CI plugin does not require the installation of any additional frameworks. The only thing that’s required is a basic configuration of the settings of a user-preferred CI tool.

Once you’ve downloaded the CI integration plugin, just add the path to it in a selected CI client instance. A few parameters including projects list, user email, user password, browser list and Screenster host address must be specified. Once this is done, you’re good to start with CSS regression testing automation on your CI server.

Screenster brings the following benefits to your visual regression testing process:

• Scriptless recorder: no coding needed to start with your visual tests. Screenster captures each user action during recording and then repeats it during a playback.
• Visual and DOM baseline: DOM model and actual screenshots are stored for a selected web UI after the test is recorded. Screenster will check for differences to the baseline during the future runs.
• Smart CSS selector: Screenster detects web page elements by capturing their CSS IDs. Selector can be manually updated in a click. Screenster easily processes AJAX and can automatically detect dynamic areas.
• Easy maintenance: you can edit tests, add or delete steps, bulk override URLs of multiple tests at a time, export and import them and even call a test within another test.

Screenster is a live proof that it’s possible to have a universal solution: make seamless integration with all major CI solutions and then run UI automation testing after each new build. So, why not check it out yourself?