I.
People love and hate End-to-End testing and for valid reasons. I’ve seen many projects (including my own) get fascinated with automated End-to-End testing and gradually come to a point where the test cases become flaky, slow and totally ignored. Let’s see why and how to make E2E testing both a good developer experience and a solid firewall for software regressions.
For the purpose of the demo, we will be testing a React\Node.js application which is a recruitment platform for HR agencies. By the end of the article we are going to have these test cases implemented:
- User can sign up
- User can login
- User can post a job ad
- User can see a candidate appear on a job board
Use Cypress.io
For quite a while there is been one major player in the field of E2E web application testing — selenium. Most of the solutions out there were basically building their APIs on top of selenium, so they all suffered from the same problem that selenium has. Cypress.io is a new player that has no selenium dependency and is set to address the shortcoming of its predecessor.
Let’s see what Cypress API can look like with our test cases.
describe('Smoke tests', () => { it('User can sign up', () => { cy .signup() .get('body').contains('Create Your First Job'); }); it('user can login', () => { cy .login() .get('body').contains('Create Your First Job') });});
OK, there is no magical signup() or login() methods, but there is a nice API for extending the ‘cy’ global with custom methods:
Cypress.addParentCommand("login", (email, password) => { cy .visit('/') .get('form input[name="email"]').clear().type(email) .get('form input[name="password"]').clear().type(password) .get('form button').click() }); Cypress.addParentCommand("signup", (name, email, password) => { cy .visit('/') .get('body').contains('Sign Up').click() .submitSignupForm(name, email) .followLinkFromEmail() .submitProfileForm(name, email) .get('body').contains('Create Your Company') .submitCompanyForm() .get('body').contains('Add Team Members') .get('button').contains('Skip').click() .get('body').contains('Create Your First Job') });
Design initial state for every test case
If we are to make the testing fast, we will need to start every test case from a predefined state of the application. Let’s define the initial states for every test case:
- “User can sign up”. We don’t really need any user-related data in the database. Though there can be some read-only data present to support the application. Let’s call it “empty” state.
- “User can login” and “User can post a job ad” both suggest that user has already undergone the signup flow, so the minimal initial state is — “signed-up”
- And finally “User can see a candidate appear on a job board” needs a job to be present, hence “job-posted” state.
So, let’s update our test cases to explicitly define the states:
describe('Smoke tests', () => { it('User can sign up', () => { cy .state('empty') .signup() .get('body').contains('Create Your First Job'); }); it('user can login', () => { cy .state('signed-up') .login() .get('body').contains('Create Your First Job') });});
The state function makes an XHR request to the API that resets its state to some predefined state.
Cypress.addParentCommand("state", (...states) => { cy .request({ url: `${Cypress.env('BASE_API')}/integration/state`, method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ states: states }) });});
We do need some support code to assist in setting the state, but the effort involved pays off in performance and maintainability of your tests. On the backend we are using MongoDB native, so the code in question can look like this:
const stateLoaders = { 'empty': require('./states/empty'), 'signed-up': require('./states/signed-up'), ...}; export const loadState = async (db, states = ['empty']) => { await clean(db); for (let state of states) { //many states? well sometimes you need to test complex
One can argue why have many states when you can have one big state for all cases. The answer is maintainability and performance. First, you save a lot of time on not loading the data that you don’t need. But what’s more important is maintainability. You application state that you are going to need for testing may conflict with each other.
For example, you want to test a case where user submitted a sign-up form but did not verify his email, so you will need a special user for that, and now we have 2 users in your database and you will have to differentiate between them in your tests. You will quickly notice that the amount of data in your state is hard to reason about. Whereas if you choose to run test case against a minimal possible state, it is easy to track state changes.
II.
In the first part of this article we looked at how using Сypress and choosing the right mocking strategy helped us write End-to-End tests that are both performant, reliable and easy to work with. To get the feeling of the snappiness check out this video that Cypress recorded for us during the test run. In this part, we will focus on another practical aspect of E2E testing — running tests on CI.
Use docker-compose
With E2E testing, we want to bring in as many parties (micro-services, APIs, transport) of the application as possible, so that we can ensure the best coverage and integration. Ideally, we should be testing a production clone, but that comes with a substantial overhead for performance — we don’t want to waste time and resources on deployment for every single build. What we want is to give a fast feedback to a developer if his commits introduced regressions or not. Here comes Docker. Docker-compose gives you this ability to declaratively bring all micro-services that your application needs together, run them on CI along with your tests.
Let’s assemble our application for testing in one nice docker-compose.yml file. For the demo we are still using this typical React/Node.js app consisting of 4 images here:
-
frontend — is the react app with a server that serves static files
-
API — is the Node.js API
-
MongoDB — persistence
-
Cypress — is our test runner that will open frontend image URL in the browser, but can also send requests to API to reset the state of the application
#docker-compose.ymlversion: '2'services: cypress: build: context: . dockerfile: Dockerfile.cypress links: - frontend - api command: /app/wait-for-it.sh frontend:3000 -t 60 -- npm run test frontend: environment: - NODE_ENV=integration build: context: . dockerfile: Dockerfile.frontend ports: - 3000:3000 expose: - 3000 links: - api command: /app/wait-for-it.sh api:4000 -t 60 -- npm run start api: environment: - NODE_ENV=integration image: 'noviopus/api-dev:latest' ports: - 4000:4000 expose: - 4000 links: - mongodb mongodb: image: mongo:3.2
Some notable things to note here.
Firstly, we are using the latest version of the API image here. The main idea is that API is developed and deployed in a backward-compatible way in regards to the frontend, so when a new version of API comes out, we know that all our deployed frontend will continue to work (within the specific environments). This allows us to evolve application without resorting to versioning of the builds.
Secondly, we are explicitly waiting for image’s dependencies to be ready to accept connections using this simple yet useful script so that we know that all services are ready before we run the first test.
Here is what Circle CI 2.0 configuration file look like with Docker-compose:
version: 2jobs: build: docker: #run all commands in this image: - image: dziamid/ubuntu-docker-compose #ubuntu + docker + docker-compose - checkout - setup_remote_docker - run: #need to login so we can pull private repos from hub in the following runs name: Login command: docker login -u $DOCKERHUB_USER -e $DOCKERHUB_EMAIL -p $DOCKERHUB_PASSWORD - run: name: Build command: docker-compose -p app build - run: name: Test command: docker-compose -p app run cypress - run: name: Collect artifacts command: | docker cp app_cypress_run_1:/app/cypress/artifacts $(pwd)/cypress/artifacts when: always #execute this run command on success or failure of previous run - store_test_results: #expose test results so you can see failing tests on the top of the page path: cypress/artifacts when: always - store_artifacts: #expose video and screenshots from cypress path: cypress/artifacts when: always - run: name: Deploy command: | # deployment is out of scope of this article
Running `docker-compose -p app -f bundle.yml run cypress` shows the glory of Docker-compose. This command will:
- start Cypress image and attach to its output
- find all dependencies of the Cypress image and start them in the background
- when the process in Cypress image will exit, it will gracefully terminate all the processes in the background
- after all the processes terminate, you can access
By exposing tests results you will see the summary of your tests on top of the page.
In the result, we have integrated E2E into the development workflow. Now we can evolve our micro-services and be confident that they can integrate with each other and the most critical application flows are working as expected.
Written by Dziamid Zayankouski
Want to learn more? Check out here