4 Automated End to End Testing Solutions Compared

Time to add some end to end tests for your next application.

I evaluated a few browser automation frameworks for end-to-end application testing and thought to share things the current landscape of tooling. This is written from the perspective of team heavily using JavaScript as the development language.

An End to end test is also known as functional-test or happy-path test which automates the sanity or spot checks before or after a software release.

I needed to pick one considering ease of development, long term maintainability, test framework integration. Let's explore a few interesting players out there.

1. The good old Selenium
2. The Chromium based browser automation API Puppeteer
3. The most hyped Cypress
4. Taiko - The new kid on the block (Its ok if you have never heard of this. I didn't too)

TL;DR; I ended up picking Taiko for its ❤️beautiful API and it just works like magic!

1. Selenium ❌

Selenium was built in 2004. The major pain point of Selenium is the architecture.

We do not enjoy coding anything that behaves differently in each run, also termed as flakiness. Let's see why Selenium tests are flaky.

The command has to transfer through multiple layers. This means there are multiple failure points. A very common scenario I have experienced is the selenium server fails to start. It is an external dependency to your tests. We need to also make sure this is up and running throughout the test. The general wisdom says systems and network are unreliable.

So let's see if there is a better way to accomplish the task.

2. Puppeteer ❌

Fast forward from 2004, 14 years later, Google released v1.0 of Puppeteer.

One thing you will notice from the previous diagram is there are fewer layers between the controller and the browser. This is the key difference between the modern end to end testing frameworks.

Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol.

This is powerful. It uses the DevTools protocol, the same low level protocol which the Chrome Developer tools uses to interact with the browser.

A code example:

const browser = await puppeteer.launch();

const page = await browser.newPage();
await page.goto('https://example.com');

console.log(await page.content());
await page.screenshot({path: 'screenshot.png'});

await browser.close();

However, it is geared towards browser automation and the end to end web application testing is only one of them. This means you have to explicitly wait for a page to load, assert things you care about which will soon make writing and maintaining tests not so fun.

3. Cypress ❌

Cypress automates the browser with its own unique architecture. Cypress takes a less traveled and hard path to make runs the test from inside the browser and communicates the actions to an outside Node process which makes testing very capable.

It comes with electron baked in. Many times, I found very simple tests work in electron which they bundle by default however fails to run in Chromium or Chrome in a headless environment.

Rave reviews, Podcast talks, Beautiful documentation, however it does not live up to the high expectations you expect.

I gave it enough energy and wanted this to work. Wanted this to work so badly. However, realized it has dangerous browser proxy mods which fails to set the redirections when run inside a corporate proxy environment and the most annoying limitation, if you are on one domain you cannot visit another domain.

Each test is limited to only visiting a single super-domain.

So if you have an application with a login page which is in another domain get ready to cry 😭. Cry so loud that Cypress team hears. I wish this changes in the future. I gave up.

4. Taiko ✅

Taiko is from ThoughtWorks. The same place where Selenium was born. It shines ✨.

It shines really well with its ❤️ beautiful API.

Show me some code...

await openBrowser();
await goto("google.com");
await write("Dennis Mathew Philip");
await click("Google Search");

You will notice there is no waiting for an element to appear. No selectors. It is smart to identify what input field.

✅ Beautiful documentation

✅ Integration with Jest and other popular testing frameworks

✅ No explicit wait needed. Less code

✅ Headless and CI friendly

It also has a REPL which makes the development a breeze.

A straightforward gif from the Taiko team:

I loved using Taiko. I am having fun writing end to end tests. I hope you will too.

To conclude...

End to end testing improves the confidence with which you release the software. We compared some of the free and open-source end to end testing solutions - Selenium, Puppeteer, Cypress and Taiko. If your web application does not have at least one functional test, start today.

Please find my other writings here:

• Automate your node dependency update
• One side rounded rectangle using SVG
• Visual Studio Code — Debug Mode
• I want TypeScript to succeed.
• What’s on my laptop?
• CoffeeScript’s most loved feature soon in JavaScript