The Good and the Bad of Selenium Test Automation Tool

Who doesn’t love a deep-background naming story? In the early 2000s, a company called Mercury Interactive ruled the test automation world with Astra Quick Test, its tool. It wasn’t the best product imaginable. For instance, you could only run tests on Windows computers, only a small number of browser versions were supported, and you had to use a single language to write scripts — VBScript. When software engineer Jason Huggins built a new, innovative test automation tool, he named his creation by referencing the chemical element and supplement recommended as an antidote for mercury poisoning — selenium. And this tool is what we’re going to review today.

Introduction to Selenium

Selenium is a suite of open source software testing automation tools that’s become the de facto product in the quality assurance world. With a list of several programming languages, all main operating systems and browsers supported, Selenium is currently used in production in such companies as Netflix, Google, HubSpot, Fitbit, and more. The whole suite provides a range of solutions for different testing problems and needs. Further in the article, we’ll talk about these tools, and the reasons Selenium stayed relevant after ten years since its creation. But for now, let’s examine what it is about Selenium that made it such a big deal.

Jason Huggins’ 2004 creation was a JavaScript framework aimed at freeing its creator from repetitive manual testing. The product that was first predictably named JavaScriptTestRunner could execute tests directly in a browser, drive interactions on the page, and rerun them without manual input. The tool took off after Huggins realized its potential, made it open source and renamed it Selenium Remote Control. The innovative part of it was that no other tool before it allowed testers to talk to the browser in their programming language of choice.

However, it soon became apparent that browsers applied security limitations on JavaScript, making it impossible to use the tool’s full capabilities. At that time Google was an avid Selenium user, but engineers struggled around the limitations. One of them, Simon Stewart, started to work on a product that would speak to browsers natively, which he called WebDriver. Selenium combined forces with WebDriver and changed the way we’ve been testing software for over a decade.

To start your Selenium journey, check out the following mini starter pack:

Selenium Suite Infrastructure

The first product in the Selenium lineup was Selenium Remote Control (now known as Selenium 1). Due to its aforementioned limitations and the following merge with WebDriver, it was soon deprecated and is no longer supported. The whole Selenium infrastructure can be visualized via the image below and we can highlight three main products that are still heavily supported and recognized by the community: Selenium WebDriver, Selenium IDE, and Selenium Grid. Let’s quickly describe each of them.

Selenium Suite today

Selenium WebDriver (Selenium 2)

Most of the time, when people talk about Selenium, they mean Selenium WebDriver. The largest chunk of the product’s development is focused on this product.

The way test automation in WebDriver works is often compared to taxi driving. There are three participants in taxi driving and test automation: customer/test engineer, car/browser, taxi driver/WebDriver. According to this analogy, a tester commands WebDriver to interact with the elements the same way a customer gives instructions to a taxi driver. Then, WebDriver gives the browser (the car) commands that sound something like this: When the button is clickable, click the button. Then the browser provides WebDriver information about values and statuses of web elements, which are later sent to the script. If you want to know more about quality assurance activities, see our comprehensive software testing whitepaper.

In this article, we will mostly talk about the benefits and drawbacks of Selenium WebDriver.

Selenium IDE

This Firefox add-on is dismissed by half of the testers and celebrated by the other half. Not meant for production, it’s easy to learn and perfect for prototyping tasks. You don’t need programming skills to successfully operate in Selenium IDE as it basically records your actions in the browser and repeats them. However, numerous official and non-official plug-ins allow you to build a Frankenstein’s monster of IDE that can likely become a production-ready tool (we will talk about it in a bit). For companies that find Selenium WebDriver too high of an entry point, using IDE with some plugins on top may be a feasible solution.

Selenium Grid

Selenium Grid allows you to run parallel tests on multiple machines and browsers at the same time. The main function of this tool is to save time. If you have to run say 100 tests but you use Selenium Grid to set up four virtual or physical machines, it will take you about one-fourth of the time it would have taken if you ran these tests one by one on a single machine. Considering how scripts normally run slow on a browser, using performance-improving techniques such as parallel testing can help with the problem. You can also use it to test one application in different browsers in parallel, when one machine is running Firefox, the other — Chrome, and so on. You can create different configurations with Grid, combining different versions of browsers and operating systems. Needless to say that when used in large production environments, Grid is a huge time-saver.

Pros of using Selenium

Finally, it’s time to discuss how good old Selenium stays afloat with numerous terrific testing tools appearing on the market each year.

+ Free

Selenium is not the only automation testing tool on the market, but it’s the only free one that can compete with paid products. As you may have seen in our automated testing tools comparison, Katalon Studio is the only viable alternative but it doesn’t provide that big of a language choice, doesn’t work on Linux, and simply doesn’t have a fanbase behind it. Selenium’s open source status makes a tool with a steep learning curve a low entry point for startups and independent developers. No surprise that even bigger companies don’t rush to switch to paid options and give away a few thousand dollars plus maintenance when Selenium still holds up.

+ Integrated with Agile, DevOps, Continuous Delivery workflow

Selenium was following the Agile and DevOps narrative before these words were even put into practice. The whole nature of Selenium falls perfectly into the main principles behind Agile, DevOps, or Continuous Delivery. How exactly is this achieved?

Since it’s portable across all platforms and often doesn’t require learning new languages, Selenium allows for an unmatched flexibility
Selenium easily integrates with various development platforms such as Jenkins, Maven, TestNG, QMetry, SauceLabs, etc.
Parallel testing with Selenium Grid allows developers to receive feedback much faster and work on the changes right away instead of waiting overnight for a test pass.

+ Supports mobile testing

With Selenium, you can also test native, hybrid, or web mobile apps, though you’ll need additional software. There are two main options — Appium and Selendroid. They are both based on Selenium so that developers already versed in it can apply the same principles when testing mobile apps.

Both tools are open source and have great community support. The main difference is that Appium supports iOS, Android, and Windows devices, while Selendroid focused exclusively on Android. Selendroid also bundles with Appium so that when you test for Android versions 2.3 through 4.3, the program will automatically switch to Selendroid.

It may seem that using Appium is a more logical choice as it doesn’t limit you to one OS. However, it makes more sense to just use Selendroid if you’re planning to test Android exclusively. It also has a few nifty features such as “hot plugging” — you can plug in devices and unplug them without interrupting the tests, increasing the number of emulators or hardware devices that can be tested at the same time.

There are two more niche tools you might want to know about as well:

Robotium — a black-box testing framework for Android
ios-driver — a Selenium WebDriver API for iOS testing integrated with Selenium Grid

+ Wide range of supported languages, platforms, and browsers

The flexibility that Selenium provides is almost unmatched in the test automation world. First, we have ten supported languages, among which are Java, Ruby, C#, PHP, JavaScript, Perl, and R — all among today’s most used programming languages. It’s also the only mainstream tool that covers Linux testing. For a detailed and updated list of supported browser versions, refer to this page.

Languages, platforms, and browsers supported by Selenium

+ Huge community

iDataLabs, data-driven research company, provides an interesting insight into the software testing tools market. You can see that Selenium takes a staggering 30.59 percent market share of all software testing tools, with its closest competitor UFT taking just over 12 percent — amazing statistics.

According to iDataLabs, more than 25k companies are using Selenium

As one of the pioneers in modern automated testing, Selenium acquired a fanbase of developers in major players like Google and startups, too. The job listings for QA professionals include Selenium proficiency as one of the required skills.

There are alternatives to Selenium, which we mentioned before, but their pricing ranges from a few thousand dollars to 10k. Having an exceptional free tool at hand keeps old testers loyal to Selenium, and new ones becoming fans.

And they are becoming fans, judging by the demand for online courses. Udemy, Lynda.com, and Coursera provide several options for learning Selenium. Hundreds of step-by-step YouTube tutorials, a topic on Quora with 25k followers, and 55k questions on StackOverflow signal a large, blossoming community, which delivers more Selenium talent to the market.

+ Large library of plugins

Selenium can be extended beyond its standard functionality with a wide range of plugins. Appium and Selendroid are two of them. On this page, you will see third-party plugins that are somewhat officially endorsed by Selenium, but you can find more unsupported ones on Github. Be sure to also google “Selenium plugins <tool name>” and you may find an extension for easy integration with your favorite programming product, such as Jenkins or Eclipse.

These plugins exist not only for WebDriver but also for Grid and Selenium IDE. The latter ones are especially extensive, able to strengthen the functionally-weak tool and make it production-ready.

Cons of using Selenium

We never praise a software product without mentioning a few pain points that come with it. Selenium has some of those, too.

– Steep learning curve

One of today’s automated testing trends is codeless testing. This approach allows anyone without deep programming knowledge to perform tests. Some testing tools such as TestComplete, Ranorex, or Tricentis provide this option by building a user-friendly UI on top of the code layer and also enabling switching between two modes. Testers can still write scripts and receive detailed feedback but skip the coding part if needed.

Selenium, however, doesn’t allow for codeless testing. You need a good grasp of one of the programming languages which narrows down the pool of people that can be engaged in the testing process. Many companies, especially startups, tend to employ their best coders to write product features and engage less skilled people in automated tests. This won’t work with Selenium.

– No built-in image comparison

In the QA process, there are difficulties in automating image verification. The simple and most effective way to decide whether an image is displayed correctly on the screen is to check manually. While it’s normal for most productions to use a combination of manual and automated testing, this is a big source of bugs since the task soon becomes repetitive and testers tend to check less often.

One of the common solutions to this problem is image comparison. You have a mockup of your image that the program should compare to the one displayed during the test. TestComplete, Ranorex, Katalon Studio, along with some others already have this functionality included. To do this in Selenium, you need a third-party software. Sikuli is a common choice. It’s an image-based recognition tool that integrates well with Selenium but it has limitations — it has to be running on the same virtual or physical screen and doesn’t recognize animations.

– No tech support

Searching for help with Selenium is a double-edged sword. Above, we mentioned the abundance of tutorials, questions, answers, and links to chat rooms where you can ask the community for help. The bad side of this is that it’s the community that gives help, not the product creators. Finding a solution to your unconventional problem or help with custom requests is possible, but difficult. In case of need, you can contact consulting or service companies for commercial support. Selenium provides a short list of such companies on their ecosystem page.

– No reporting capabilities

Testers, developers, project managers — they all need access to test results, preferably visualized in the form of charts, tables, and supported by screenshots. What’s the point in writing scripts and performing tests when the final information can be lost or not communicated to the people who need it the most?

The lack of automatically generated reports is one of the biggest challenges of Selenium. To capture test failures in Selenium, you must take a screenshot at the moment of failure. This is far from the readable format that the team needs to quickly diagnose the problem. While different vendors offer reporting functionality with data-driven insights and team-working tools integration, Selenium has to rely on third-party solutions. The following are the most popular ones:

TestNG creates two types of reports upon test execution: detailed and summary. The summary provides simple passed/failed data; while detailed reports have logs, errors, test groups, etc.
JUnit uses HTML to generate simple reports in Selenium with indicators “failed” and “succeeded.”
Extent Library is the most complex option: It creates test summaries, includes screenshots, generates pie charts, and so on.
Allure creates beautiful reports with graphs, a timeline, and categorized test results — all on a handy dashboard.

Final word

What would happen if Selenium suddenly announced it’s going the paid route? It’s hard to critique something when it’s provided to you completely free, but how painful would the pitfalls be if we suddenly had to pay for them?

Selenium is not perfect but it’s just good enough for testers to put forth a greater effort and ignore the codeless trend. It hits that sweet spot when you’re prepared to spend a few hours looking for a solution online instead of getting the help you paid for. And it still has a few aces up its sleeve that fit right in with today’s engineering narrative: faster delivery, coming in small sections to handle easily, receiving immediate feedback.

Selenium is a universal use case. It doesn’t overshadow all of the other test automation tools, especially if you have the budget for them. But its monopolizing nature makes you think twice before you look the other way. Perhaps, it’s not that bad.

This post is a part of our “The Good and the Bad” series. For more information about the pros and cons of the most popular technologies, see the other articles from the series:

The Good and The Bad of Xamarin Mobile Development

The Good and the Bad of JavaScript Full Stack Development

The Good and the Bad of Node.js Web App Development

The Good and the Bad of ReactJS and React Native

The Good and the Bad of Swift Programming Language

The Good and the Bad of .NET Framework Programming

The Good and the Bad of Angular Development

The Good and the Bad of Java Programming

Originally published at AltexSoft Tech Blog “The Good and the Bad of Selenium Test Automation Tool”