A while ago, a friend of mine, who is just beginning to explore the wonderful world of frontend development, asked me how to start testing her application. Over the phone. I told her that, obviously, I can’t do it over the phone as there is so much to learn about this subject. I promised to send her links to guide her along the way.
And so I sat down on my computer and googled the subject. I found lots of links, which I sent her, but was dissatisfied with the depth of what they were discussing. I could not find a comprehensive guide — from the point of view of a frontend newbie — to testing frontend applications. I could not find a guide that discusses both the theory and the practice, and is oriented towards testing frontend application.
So I decided to write one. And this is the third part of this series of blogs. You can find part one here, and part two here. Also, for the purpose of this blog, I wrote a small app — Calculator — that I will use to demonstrate the various types of testing. You can see the source code here. Note that it isn’t yet fully tested, but it will be once this series of blog posts will be done.)
End To End Testing
In part II, we’ve seen one end of the spectrum of testing — unit testing. We tested the core logic of the application, the module calculator, using mocha and this test module.
In this part, we’ll see the other end of the spectrum of testing — end to end testing, where we test the whole application together, and test it as a user would — essentially automating whatever the user does.
In our case, the calculator front-end app is the whole app — there’s no backend support, so in our case E2E testing means opening up the application in a real browser, doing a set of calculations using the keypad, and ensuring the value in the display is the correct one.
Do we need to check all permutations, like we did in the unit test? No! We already checked that in the unit tests. What we check in the E2E tests is not that this unit works, or that unit works, but rather that they all work together.
How Many E2E Tests
This is one reason why there should not be many E2E tests — if we tested correctly in the unit and integration tests, we have probably checked everything. The E2E test should just check that the glue binding all units works.
There is two more reasons why there are not many E2E tests. The first reason is that these tests are slow. Having hundreds of them, like you would have in unit and integration tests, means that running your tests would take a long time, and that is something that is imperative — tests should run fast.
The third reason to not have many E2E tests is that these tests tend to be flaky. Flaky tests are tests that usually pass, but sometimes fail. This cannot (usually) happen in unit tests, as they are usually simple input/process/output and involve mostly the CPU. But the more a test involves I/O (where in I/O I mean anything that is not CPU or memory driven), the more flaky it becomes. Can we mitigate the flakiness? Yes, to a bearable amount of flakiness. Can we kill flakiness entirely in E2E tests? Maybe, but I’ve never seen it happen.
So to remove the flakiness from our tests, have as few E2E tests as possible. Have one to 10 tests, each one testing the main flows of your application. Don’t try to do more with the tests, unless you absolutely have to.
Writing Frontend E2E tests
OK, already. Let’s get down to the business of writing frontend E2E testing. For our frontend E2E, we need to run two things: (i) a browser, and (ii) a web server that serves our frontend code.
Since we use Mocha for our e2e tests, just like for our unit tests (they all run together!), we will set up the browser and the web server using Mocha’s before hook, and clean them up using Mocha’s after function. The before and after hooks run before and after all the tests are run, and are used to setup up stuff that the tests functions themselves can use.
Let’s first look at setting up a web server.
Setting Up a Web Server in Mocha
A web server in Node? Immediately, express comes to mind, so without further ado, let’s see the code in our test:
<a href="https://medium.com/media/16d357d4cfeb8ef32746241e106b3e84/href">https://medium.com/media/16d357d4cfeb8ef32746241e106b3e84/href</a>
In the setup, in the before hook, we create an express app, point it to our dist folder, and make it listen on 8080. The teardown, in the after hook, closes down the server.
What’s this dist folder? Well, it’s where we bundle our JS files (using webpack), and to where we copy our HTML and CSS. You can see that we do this in the npm build script in the package.json:
{
"name": "frontend-testing",
"scripts": {
"build": "webpack && cp public/* dist",
"test": "mocha 'test/**/test-*.js' && eslint test lib",
...
},
This means that for e2e tests, we need to remember to npm run build before we npm test. See how inconvenient that is? We didn’t need to do that for the unit tests because they run under node and don’t need transpiling or bundling.
For completeness’ sake, let’s look at the webpack.config.js that tells webpack how to do the bundling:
module.exports = {
entry: './lib/app.js',
output: {
filename: 'bundle.js',
path: path.resolve(__dirname, 'dist')
},
...
}
Webpack will read our app.js and bundle all the files it requires (recursively) into the bundle.js in the dist folder.
This dist folder is used both in production, and as we saw, in our E2E tests. This is important — E2E tests run in an environment that is as similar to production as possible.
Setting Up a Browser in Mocha
Now that we have the backend setup, and our application is served, all that is left is to run the browser so we can start to drive it on our application. Which package shall we use to drive the automation? I typically use selenium-webdriver, as it is a popular package.
First, let’s see how we use the driver, before we understand how to set it up:
<a href="https://medium.com/media/78ac7b81d4399d75347504042b5c8794/href">https://medium.com/media/78ac7b81d4399d75347504042b5c8794/href</a>
In the before, we prepare the driver, and in the after, we clean it up. Preparing the driver will run a browser (Chrome, as we will shortly see), and cleaning up will close the browser. Note that preparing the driver is asynchronous, and returns a promise, so we use the new async/await functionality to make the code look nicer. (Hurrah for Node 7.7 — the first version to natively support async/await!)
And, finally, in the test function, we browse to the site http://localhost:8080, again using await, given that driver.get is an asynchronous function.
So how do those prepareDriver and cleanupDriver look like?
<a href="https://medium.com/media/93c904df607a02f33fec97eb525a0db0/href">https://medium.com/media/93c904df607a02f33fec97eb525a0db0/href</a>
Oh, my. This is heavy-duty stuff. And I have to admit something — this code was written in blood. (Oh, and it works only in Unix systems — if any of you Windows people want to make it work in Windows, I’ll be happy to accept a pull request…). That means that it was written through a combination of Google/Stack Overflow/webdriver documentation, and heavily modified by experience. But it works!
Theoretically, you can just copy/paste it into your tests, without understanding, but let’s dig into it for a second.
The first two lines bring the webdriver, and it’s companion browser driver. The way Selenium Webdriver works is by having an API (in the module selenium-webdriver that we import in line 1) that works with any browser, and it relies on browser drivers to… drive the various browsers. The driver browser I used is chromedriver, imported in line 2.
The chrome driver does not need Chrome on the machine — it actually installs its own Chrome executable when you do npm install. Unfortunately, for some reason that I cannot understand, it can’t find it, and the chromedriver directory needs to be added to the PATH (this addition to the path is what doesn’t work in Windows). This we do in line 9. We also remove it from the path in the cleanup phase, in line 22.
So we’ve set up the browser driver, and it’s time to setup (and return) the web driver — which we do in lines 11–15. And because the build function is asynchronous and returns a promise, we await it.
Why do we do those things in lines 11–15? Why those exactly? That’s part of what’s written in blood. The reasoning is shrouded in the mists of the past, and cloaked in the mantle of experience. Feel free to copy/paste — no guarantees attached, but I’ve been using this code for a while now with no problems.
The Test! The Test!
We’re done setting up — it’s time to look at the code that uses the webdriver to drive the browser and test our stuff.
Let’s bring the test code here, but let’s do it slowly, because it’s a mouthful.
<a href="https://medium.com/media/4161dd5c8ec50053232f7e730574e1ff/href">https://medium.com/media/4161dd5c8ec50053232f7e730574e1ff/href</a>
We’re skipping the setup, which we saw earlier, and diving into the test function itself.
The code here browses to the calculator app, and checks that the title is “Calculator”. Let’s see this:
Line 1 we already saw — we ask the driver to browse to our app. And we do not forget to await it.
Let’s for a second skip to line 9. In it, we ask the browser to return to us the title of the browser (await-ing the answer because it’s asynchronous), and then, in line 10, we check that it has the correct value.
So why do we retry this, using the promise-retry module? The reason is very important, and we will see the same thing happening in the rest of the test — when we ask the browser to do something, like navigating to a URL, the browser will do it, but asynchronously. Don’t let the await fool you! We are awaiting for the browser to say — “OK, I’ll do it”, and not for the operation to end.
Unfortunately, if you do not retry in this case, it could still work, because the browser is very fast. But try running it in a CI system like Travis, as I did, and you may get a failure, because CI-s are usually slower than your local system.
And that is why, in E2E tests on the browser, we need to retry our checks.
Looking at Elements
Onward and upward to the next part of the test!
<a href="https://medium.com/media/7ec396ee22ee616bf3ef627924314583/href">https://medium.com/media/7ec396ee22ee616bf3ef627924314583/href</a>
The next thing we check is that, initially, the display is “0” (that’s a zero…). To do that, we need to find the element that holds the display, which in our case has the class display. This we do in line 7 — webdriver’s findElement returns the element we look for. And we can look for By.id, or By.css, or some other stuff. I usually use By.css — which accepts a selector and is very versatile, although By.javascript is probably the most versatile.
(and if you hadn’t noticed, By is imported from selenium-webdriver at the top).
Once we have the element, we can use getText() (you can use other functions on elements, too) to get at the text and check its value in line 10. And don’t forget:
Driving the UI
It’s time to drive the application — to click on the digits and the operators, and to check that the calculation is as we expect:
<a href="https://medium.com/media/02903af84447bc5db8a29af895d9f2cb/href">https://medium.com/media/02903af84447bc5db8a29af895d9f2cb/href</a>
There we go — we first find the elements that we want to click, in lines 2–4. Then we click on them, in lines 6–7 (the calculation, if you can’t follow, is “42*2=”.
Then we retry until we get the correct value — “84”.
Running The Tests This Far
So we’ve got E2E tests, and unit tests. Let’s run them all using npm test:
Running E2E and unit tests together
We’re green!
Some Words on the use of await
If you look at a lot of samples out there on the web, you will see they do not use async/await, or even wait on the result using promises. Nope, they write code that is synchronous. How does this work? Honestly, I don’t know, but it looks like some weird shenanigans going on inside the webdriver. And, as even selenium says, this was an interim solution, until Node gets async/await support.
Well, guess what happened?
Some Words on the Documentation
Selenium’s documentation is, well, Java-ish. And that was not a compliment. On the other hand, the information is there. So bear with it, after a few tests, you will get it.
Next Week
We’ve seen the tests at the other end of the spectrum of tests — E2E tests that test it all. Next week, we’ll go back to the middle.
Summary
What did we see this week?
- We saw hairy code that sets up the browser. Luckily, we need to write it only once.
- We saw how to use the webdriver API to drive the browser and get at DOM elements and their values.
- We used async/await extensively, as all the webdriver API is asynchronously using promises.
- We understood why we need to retry stuff in E2E tests.