This article introduces an idea of stateless end-to-end tests using Google Chrome’s Lighthouse and automatic snapshots. This approach enables fast development experience and tests availability, performance, and quality regressions. For practical examples the article uses Treo.
Before we dive in, let’s define end-to-end tests, why they are valuable, and why they are hard to develop.
Testing Pyramid (source Martin Fowler, triangle authorship Kent C. Dotts)
End-to-end tests evaluate a system by using it as a real user would. They are slower and more expensive compared to unit tests. But they bring the most confidence by validating the entire chain of system dependencies.
Let’s analyze both challenges with E2E tests.
End-to-end tests are slow. It takes time to start new browser instance and load the page. It may take 20 seconds for one test, and with many tests executed in sequence, it goes up to minutes.
E2E tests are expensive or hard to develop and maintain. To start, a developer has to configure selenium or another tool. Deal with a slow feedback loop and cryptic error messages. Deploy to CI, which is also quite challenging. In a month, these tests start to fail, and no one understands how they work and how to debug them. I saw this story repeats itself over and over in different projects.
We need something better. Which trade-offs should we choose to overcome these challenges? How can we make end-to-end tests simple and fast, but still valuable?
Stateless end-to-end tests
In my experience, 80% of the value of end-to-end tests comes with initial page load and checking data on the page. Real users interact with production environment using a browser. So if test from end to end, it has to be an actual deployment and a real browser.
Definition: the stateless end-to-end test does not mutate state allowing parallel execution and an actual deployment validation.
Main idea: describe a web app as a set of publicly available URLs, perform Lighthouse audit for each URL, compare results with a previously successful run to detect regressions.
If we don’t reset an environment for each test, we can execute tests in parallel. It improves feedback loop dramatically and allows to use production/staging deployment.
If we test an actual deployment, we can guarantee the full system availability. (web servers, CDN, databases, third-party services)
If instead of writing manual expectations, we use Lighthouse (and experience of 100s of developers) to evaluate the page properties and save it as a snapshot. Next time we deploy a new version, we can ensure the page didn’t change compared to previously correct evaluation.
It’s highly valuable to know, that after new deployment all critical pages are available and behave as they used. The user can open the page and start to interact with it. To keep the whole experience under control, monitor runtime exceptions (with Rollbar or Sentry) and fix them immediately.
No kind of testing can guarantee 100% error proof. We need to keep a balance between development effort and value. By using stateless end-to-end tests, we get a great value with minimal effort. In my experience, this approach works nicely for small and medium-size projects. Large-scale projects are usually invested heavily in monitoring, QA, and custom processes.
Examples using Treo
Treo is a cloud infrastructure for Lighthouse that provides simple end-to-end testing flow and integrates with Slack & Github.
With Treo end-to-end testing setup is just a few clicks. To execute a test, just call an API and receive status update to Slack or Email in less than 20 seconds.
A testing flow with Treo
Personal website example: alekseykulikov.com
Let’s look at a typical personal website: https://alekseykulikov.com. It contains three pages: Home, Blog, and Not Found.
Typical personal website with three pages: Home, Blog, 404
Why test this simple website? Because I develop it locally and deploy with git push. These are three steps: build, upload to S3, invalidate CloudFront distribution. I want this site to be available, performant and accessible. I don’t want to validate deployment manually, by clicking and running Lighthouse audits for each page. Just push and forget.
Step by step tutorial (video):
- Log in to https://treo.sh using Github account (free trial is ~500 audits).
- Click “Add new test suite” and name it;
- Add three pages with title and URL;
- Click “Create”, wait a few seconds, and look at results;
- Copy curl command to deploy script with an automatic message from Git.
MESSAGE=`git log --format=%B -n 1 $sha1 | xargs`curl https://api.treo.sh/v1/suites/1 \-H "Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \-d message="$MESSAGE"
Next time, I do changes and break accessibility on /blog, Treo notifies me:
Failed Report
Web application example: treo.sh
Treo uses itself for testing. It is a SPA with 16 different pages. The testing process is similar to the first example with a few advancements. It uses cookies for authentication and special pages for performance monitoring in US/EU regions.
Examples of treo.sh UI
It’s quite natural to describe a SPA with a set of URLs. Usually, each URL is a router entry. Each entry is a separate view with custom logic. Loading the view without a difference from the previous run, gives a solid guarantee, that it continues to work.
During development of Treo, I caught a lot of bugs using Treo. Broken pages, failed Webpack builds, accessibility and performance regressions. That gives me confidence about this approach. Stateless end-to-end tests are the most useful kind of tests for an invested amount of time. Next, go to unit tests, but this is entirely different story.
I’m looking forward to hearing feedback from the community! Let’s discuss an idea of stateless end-to-end tests on twitter @alekseykulikov_ or in comments.
Once again, it is not a universal approach. It is a set of trade-offs to achieve better quality and faster release cycle for small and medium-size projects.
If you need any support with configuring Treo, contact me: [email protected]. I’d be happy to help your team to define most important pages and look to Lighthouse results.
Let’s build reliable web apps. Happy testing! 👍