Two pillars of automated testing

When your team has been asked to move fast and break things, methodological purity can fast give way to sprint-speed pragmatism. However, this phrase — once the credo of Facebook — doesn’t mean we can take risks in the QA department. It’s really an admonition for us to open our eyes to new opportunities instead of closing them and praying that things will stay the same. A kind of rallying cry to be disruptive and try (and discard, if necessary) new things quickly.

What’s all this got to do with testing and QA?

Quite a bit. In our experience, QA is one of the slowest moving disciplines across the development landscape. This goes for both the supply side and the demand side of the market. It’s rare for new testing technology to reach the market in the first place. And equally, it’s not so common for product teams to have the capacity, time, and will to look for new tools. This is not surprising since many QA teams have a corpus of test libraries that they can’t simply forget about.

All of which is by way of encouragement for you to get to know a testing framework called Cypress, which is ruffling a few feathers of Selenium, the market leader. You might have heard of it. Timewise, Cypress has just left Beta, and we thought that this would be an opportune moment to contrast the very different approaches the two frameworks take. In this post, we’ll review their main differences, most of which stem from a smaller set of design choices made by developers.

A crash course in history

Selenium was originally developed in 2004 by Jason Huggins and has been maintained and updated to cope with technology changes ever since then. Along the way, a super team coalesced around the technology. Cypress is developed by Brian Mann and his no-less-super team, and its first commit was just over four years ago. Both Selenium and Cypress are now open source.

Architecture

Architecturally, the two are polar opposites, and it’s these architectural differences that are likely to inform your preferences rather than anything else.

Selenium is composed of a component called WebDriver, which can manage and control a (specific) browser, together with a set of bindings that determine which language your tests are written in. When tests run, at least three processes are involved:

• WebDriver
• A browser driver
• The browser itself

The network overhead between these processes has been known to make test execution slow. Even if you have everything running on the same machine, that network overhead persists — locally, Selenium uses the 127.0.0.1 loopback interface.

Cypress takes a different approach and comes as a single monolithic application — the only choice you need to make is whether you want to install it via npm or in standalone mode. Choosing npm gives you access to the rest of its ecosystem inside your test suite — indeed behind Cypress is a Node.js server process (which, in standalone mode, is embedded). Most people choose the npm route, but standalone mode might suit a brief visit while evaluating it.

Cypress runs tests inside the browser itself and has access to the DOM and even the application under test, a configuration which sets things up, among other things, for it to be blisteringly fast. Don’t forget though that having that back-end Node.js process means that it can also work outside of the browser for tasks that require a higher privilege. This said it does take a little extra work to communicate with back-end processes like a database or another server.

This trade-off means that Cypress can only support JavaScript, and will never support anything else, while Selenium currently has bindings for Java, C#, Ruby, Python, and JavaScript (node), which is currently in Alpha. And for some, this is going to be a killer feature and may well be the reason why some teams are going to have to reject Cypress out of hand.

Flake

Flaky tests — tests that could pass or fail when in exactly the same application state — are the bane of testers and QA engineers. Cypress wraps all DOM queries with the retry-and-timeout logic that its developers claim to work better for real web apps, leading to a massive reduction in test flake.

The Cypress team also claims that despite Selenium’s impressive evolution to support today’s web, Cypress’s support for modern asynchronous single page applications makes it a more natural fit for current development workflows and application design patterns.

Code, libraries, and on-boarding

Both Selenium and Cypress mandate that you write your tests in a programming language. This might sound obvious but there are some testing products that use other models, for example, via GUI-based test recorders. In Cypress’s case, that language must be JavaScript. Language wars aside, Selenium offers a choice, and is, correctly, more of a library than a framework like Cypress. Selenium goes on to integrate well with many unit testing frameworks and assertion libraries, while with Cypress, the choice is limited to Mocha, Chai, and Sinon.

Out of the box, it’s, therefore, easier to get going with Cypress, because everything is already bundled. Selenium takes a little more preparation and thought about which companion technologies are the best fit for your use case. It’s not a huge speed bump but it might make a difference, especially when you’re trying to assess multiple options.

Debugging

Cypress is aimed squarely at developers and QA engineers and has a great debugging story. Because it executes in the same run loop as the application, it has access to variables, code, and the things the browser makes available like document, window, and debugger. Once you’re familiar with its asynchronous nature, it’s a simple job to hop into the debugger. It also takes automatic snapshots of the DOM and lets you “time travel” to earlier application states if you’re hunting down a bug in your test scripts.

With WebDriver/Selenium, debugging is not particularly intuitive. When you hit problems, all you get is an HTTP error response after running all the tests. At this point, your application is gone and the browser is closed — yet you somehow have to intuit the cause of that HTTP error. It can be frustrating.

The bottom line

Words and laundry-list comparisons don’t seem to do either of these great testing solutions justice. You have to experience them in the context of your own needs. What to some might be a showstopping lack of a feature or an architectural quirk, might to others not make any difference at all, or even be a benefit.

Sure, we’re excited that the arrival of Cypress means that for the first time in many years, there’s another open-source automated testing game in town. But there’s no doubting that Selenium is a wonderful success story and will remain vital to many for the foreseeable future — it mitigates several gaping holes in Cypress’s current capabilities, such as cross-browser testing and support for multiple languages.

If your team uses Selenium, we can’t tell you if you should switch to Cypress, but what we can tell you is that you should beg, borrow, or steal the time to check it. Then, try it out for real perhaps on a small, well-defined new product feature, rather than your complete application. Maybe, ultimately, the smart decision will be to use both technologies?

Go on. Move fast. Break things.