Running End to End tests as Google Cloud Functions

It started all here: Google has announced support for browsers in their Cloud Functions service. But well, just for one browser. Guess which one? You are right, it’s Google Chrome. A headless Chrome. Nevertheless, this announcement is huge, this can be the beginning of revolution for browser testing as we know it.

Announcement post

Let me elaborate. Cloud Functions in Google (similarly lambdas in AWS) are serverless stateless functions which you pay only for the time of their actual execution. Inside a testing community there was a huge interest towards using them. Indeed, we are launching our browsers just the same way — to launch and stop, and we do not want to pay for the time test is not running! Before this announcement it was pretty hard to make browsers run inside cloud functions, but now Google installs one so we are ready to try it. Can it be the game changer?

Google recommends using Puppeteer library to control the browser. Puppeteer is not a testing framework by itself, so to write and run actual test we will use CodeceptJS testing framework. This one can run tests in various backends, like Selenium, but also has built in support for Puppeteer. If you never met CodeceptJS before, check my previous post: “Effective End 2 End Testing in JavaScript with CodeceptJS”. And today we will try something advanced: running functional test as cloud function.

CodeceptJS testing framework

At first you will need to sign up in the Google Cloud Platform. Even if you don’t have an account there it’s pretty easy to create one, as everyone has Google Account. Google will ask you for a credit card but don’t worry, you will get 300$ for a year to spend, and cloud functions are included into free tier. After the account created you will need to install gcloud command line tool.

Cloud Functions Free Tier includes up to 2 million invocations and 1 million seconds of free compute time per month

And now let’s do some JavaScript stuff. Let’s start by making package.json file including codeceptjs and puppeteer libraries:

{"name": "browserTest","main": "index.js","dependencies": {"codeceptjs": "^1.3.3","puppeteer": "^1.6.2"}}

Then run npm install . You don’t need to install any other testing framework, like Mocha, because CodeceptJS already brings one on board. It also has assertions baked-in.

Unlike in traditional CodeceptJS installation (which still can be done), we are not able to use CodeceptJS CLI runner here as Google expects pure JavaScript function inside index.js . CodeceptJS has API for creating a custom runner which we will use to run our tests. Copy-paste this snippet into index.js:

In this setup we created configuration for Puppeteer (please note that --no-sandbox option which is required for browser to be launched in cloud) and created a basic reporter. However, we didn’t create any test yet! Tests are standalone files with *_test.js suffix which are loaded from the current directory.

So let’s create one! We will do a basic test around GitHub, that’s why I added into Puppeteer config github.com as base URL value. So a test file, named github_test.js should be pretty simple, however include some interactions and assertions. Here is the one I prepared:

(I use _retry()_ for some commands as the page may not be loaded yet fully)

As you see, you can read the test and understand its output. The syntax is pretty simple, without needless CSS or XPath selectors.

That’s all our test. But let’s get back to our index.js and our main function which is named browserTest (that name will be used by Google). But before we deploy it to the cloud let’s try to run it locally with NodeJS:

node -e 'require("./index").browserTest()'

test is passed, the error is not related

If you see an output like this, the test is passing and we are ready to push it into the Cloud.

Execute this command:

gcloud beta functions deploy browserTest --trigger-http --runtime nodejs8 --memory 1024MB

On first run you will be asked

On first run you will be asked to enable Google Cloud Functions API. You need to agree on that, and please check that billing is enabled for this project. Otherwise browser won’t be able to reach external websites.

The expected output

Once the function is deployed it can be executed by the next command:

gcloud beta functions call browserTest

If everything is fine we should see this response after a few seconds:

If not — try rerunning function or look into logs, which Google also provides us with the next command:

gcloud functions logs read

Logs should contain tests output written step by step with assertion failures if there are any. If an error has happened it’s easy to detect where it was. For example, here how the error report will look like in logs:

But I hope you will get the successfully passing test, instead of this one

Finally, let’s switch from the terminal to fancy Dashboard for detailed report on cloud function executions.

Summary

The test is running and starting really fast. However, there are some limitations you need to consider before using Cloud Functions for testing. The strongest one is timeout. By default Google limits execution of cloud functions to 1 minute which can be increased to 9 minutes but no more than that.

That makes Google Cloud not yet ready for complex end to end tests with long scenarios. However, Google Cloud Functions are perfect for healthcheck tests. So you can pick the subset of your tests, and run them from time to time by Google Cloud to ensure availability of your service. If you cover the most basic functionality like login, search, etc and deploy such tests into cloud they will come in handy one day.

Other notable limitations is read-only filesystem, preinstalled Chrome, no control of the system. But the starting time is impressive and the pricing model is very interesting. I hope one day Google will consider increasing the timeout or allow us to spawn multiple functions to run our tests as Cloud Functions.