How to set up E2E browser testing for your GitHub project

Nowadays, a vast number of projects, small and large, is developed using Node.js tools and hosted on GitHub. Authors of these projects often face the need to choose a tool for end-to-end testing. When we at DevExpress created TestCafe, we wanted it to perfectly fit in the modern Node.js development workflow.

• The same language is used both in tests and projects you are developing — modern JavaScript with support for ES6 and ES7 features.
• Easy installation and launch. No need to install and maintain additional products and browser plugins. Tests can be run right after npm install.
• Easy integration with CI systems.
• TestCafe functionality can be extended with plugins.
• Tests can use any third-party Node.js module.

In this article, I’ll show how easy it is to set up automated testing for your Node.js web project on GitHub. Using a real project (Bootstrap) as an example, I’ll describe how to set up testing in local browsers on a developer’s machine and how to configure Continuous Integration testing on Travis CI step by step. In the CI testing part, I’ll show how to run tests in the browsers provided by Travis CI (Chrome and Firefox), headless browsers (the NightmareJS browser automation library) and in cloud services that provide a wide range of operating systems and browsers for testing (SauceLabs).

I’ll show how end-to-end testing can be used at each stage of the project’s workflow:

• On a local machine during the development. You can run tests in any browser without installing additional libraries and plugins.
• In a pull request. This allows you to ensure that the project works well before the changes are pushed to the master branch.
• When pushing commits to the master branch.

To run tests, we will use TestCafe testing framework.

We will write tests for Bootstrap, the most popular HTML, CSS and JavaScript Framework for developing web projects. To this end, I’ve forked the Bootstrap repository and created a new branch testcafe. When this article is written, the repository has unit tests based on QUnit that are run on sending each pull request and merging to the master branch. In addition, the repo contains pages used to test components visually, with manual verification by humans. We’ll write functional tests for one of these pages.

Writing Tests

We will use the Page Object pattern in our tests. This approach allows us to create more declarative tests. In this article, we won’t focus on test syntax. For information about test API, refer to TestCafe documentation.

Let’s write a test for a page with Bootstrap’s Modal component. Modal Dialog internally contains a number of other components (Accordion, Button, etc).

The TestCafe selector system will help us create a Page Object. We’ll describe the entire page and its components with classes in a separate page-model.js file.

You can find the entire page object here.

With the page object, tests do not have to work with HTML markup. Instead, they operate the object representation of the page.

A complete version of this test fixture is available here.

Testing in Local Browsers During Development

To test a modal dialog page, you need to have an HTTP server that will serve up this page. You may already have a mechanism that serves tested pages in your project. I will use a simple, zero-configuration http-server.

Install TestCafe and http-server.

npm install -g testcafe http-server

Immediately after this, you’ll be able to run tests in any browser installed on your machine with one line in the console.

testcafe chrome,firefox js/tests/functional/*-test.js --app "http-server ./ -p 3000"

The --app option allows you to specify a command that deploys the application you are going to test.

TestCafe will start http-server, launch the specified browsers, run tests in them, stop the server and output the report.

TestCafe CLI can also launch a browser by the path to its executable (good for portable browsers), pass arguments to the browser launch command and run tests in remote browsers.

Setting Up Continuous Integration Testing

We’ll use Travis as a CI service. Alternatively, we could use other services (AppVeyor, CircleCI, etc) with different features.

Travis CI provides Linux virtual machines with pre-installed Chrome and Firefox.

At first, we should add the modules we have installed earlier to the project’s dev dependencies. You can do this by manually editing package.json or by running the following command

npm install testcafe http-server --save-dev

Setting Up Travis CI

Let’s configure Travis CI to run tests for each pull request. First, you need to enable Travis for your project.

Sign in to Travis CI with your GitHub account. Travis CI will synchronize your repositories from GitHub. You can see them on your profile page.

Enable Travis CI for a repository you want to build by flicking the switch on.

By default, Travic CI runs builds on pushes and pull requests. You can change this behavior in your repository’s settings.

Let’s configure the .travis.yml file. In this file, we’ll specify the Node.js version and browser settings. In addition, since Travis CI uses Ubuntu Server virtual machines that do not have regular graphical environment, we set up and use Xvfb to run browsers headlessly.

For details on Travis configuration, see its documentation.

Run Tests in a Pre-installed Browser for a Pull Request

To run tests in a browser pre-installed to Travis CI, add an npm test command to package.json with the line we used to run tests locally.

Now when a pull request to the master branch is created, tests will run on Travis and you will be able to view a test run report. The commit status in the pull request will change accordingly.

After the pull request is merged and changes are pulled to the master branch, Travis CI will run tests one more time. You can view their status in your Travis CI account or you can add a badge to your repository readme.

Configure Tests to Run in a Headless Browser

If you do not need to test browser-dependent behavior, you may find it useful to run tests in a headless browser. This allows you to avoid installing real browsers on a testing machine. You only need to install a single module with npm install. For instance, a TestCafe community member created a plugin that allows you to run tests in NighmareJS out of the box. Let’s install this plugin.

npm install testcafe-browser-provider-nightmare --save-dev

After that, you can pass nightmare as a browser name to the test run command.

testcafe nightmare js/tests/functional/*-test.js

TestCafe allows you to easily create your own plugin for any such library. You can find the instructions in our documentation.

Testing in Any Browsers with Cross-Browser Testing Services

If the testing options described above do not satisfy you needs, consider testing in real browsers with crossbrowser testing services (BrowserStack, SauceLabs, etc). TestCafe has a plugin that allows you to run tests on virtual machines on SauceLabs out of the box. A SauceLabs account is free if you use it to test an opensource project.

To run tests, first install a plugin.

npm install testcafe-browser-provider-saucelabs --save-dev

Then you need to specify the login and access key for you SauceLabs account in global variables (SAUCE_USERNAME, SAUCE_ACCESS_KEY). When running tests locally, you can do this through the console. To run tests on Travis, specify the credentials in your Travis account.

TestCafe can show all browsers available via SauceLabs with the following command:

testcafe --list-browsers saucelabs

Let’s run our tests in desktop and mobile browsers:

testcafe "saucelabs:Chrome@beta:Windows 10,saucelabs:iPad [email protected]" js/tests/functional/*-test.js --app "http-server ./ -p 3000"

TestCafe will set up a tunnel between your machine and the SauceLabs server, start virtual machines, run tests and output the result.

If you are opening a pull request from a repository fork, there are certain limitations. It is not possible to do cross-browser testing in pull requests via Travis CI because this may potentially compromise your SauceLabs credentials. Fortunately, you can work around this by using Savage.

Conclusion

As you can see, configuring TestCafe does not require any difficult operations. It is made with just a single console command. TestCafe can output reports in different formats (json, nunit, spec, list, minimal) and allows you to create your own report format with a plugin. This enables you to use TestCafe with every CI system. You can also run your tests with a gulp or grunt task.

Since TestCafe is written for Node.js and has both command line and programmatic API, you can configure the testing process flexibly to suit your particular tasks and use third-party Node.js libraries.

If you are interested in TestCafe, you can learn more on our site or repository.

Please use the comments section to tell if you have any questions when building CI with the existing solutions. Do you think TestCafe may help you? I would appreciate your opinions and feedback. And I’m ready to answer your questions.