Stateless end-to-end testing for web apps using Chrome, Lighthouse, and Treo

This article introduces an idea of stateless end-to-end tests using and automatic snapshots. This approach enables fast development experience and tests availability, performance, and quality regressions. For practical examples the article uses . Google Chrome’s Lighthouse Treo Before we dive in, let’s define end-to-end tests, why they are valuable, and why they are hard to develop. Testing Pyramid (source authorship ) Martin Fowler , triangle Kent C. Dotts . They are slower and more expensive compared to unit tests. But they bring the most by validating the entire chain of system dependencies. End-to-end tests evaluate a system by using it as a real user would confidence Let’s analyze both challenges with E2E tests. End-to-end tests are . It takes time to start new browser instance and load the page. It may take 20 seconds for one test, and with many tests executed in sequence, it goes up to minutes. slow E2E tests are or hard to develop and maintain. To start, a developer has to configure selenium or another tool. Deal with a slow feedback loop and cryptic error messages. Deploy to CI, which is also quite challenging. In a month, these tests start to fail, and no one understands how they work and how to debug them. I saw this story repeats itself over and over in different projects. expensive We need something better. Which should we choose to overcome these challenges? How can we make end-to-end tests simple and fast, but still valuable? trade-offs Stateless end-to-end tests In my experience, of the value of end-to-end tests comes with initial page load and checking data on the page. Real users interact with production environment using a browser. So 80% if test from end to end, it has to be an actual deployment and a real browser. : the stateless end-to-end test does not mutate state allowing parallel execution and an actual deployment validation. Definition : describe a web app as a set of publicly available URLs, perform audit for each URL, compare results with a previously successful run to detect regressions. Main idea Lighthouse If we don’t reset an environment for each test, we can execute tests in parallel. It improves feedback loop dramatically and allows to use production/staging deployment. If we test an actual deployment, we can guarantee the full system availability. (web servers, CDN, databases, third-party services) If instead of writing manual expectations, we use Lighthouse ( ) to evaluate the page properties and save it as a snapshot. Next time we deploy a new version, we can ensure the page didn’t change compared to previously correct evaluation. and experience of 100s of developers It’s highly valuable to know, that after new deployment . The user can open the page and start to interact with it. To keep the whole experience under control, monitor runtime exceptions (with or ) and fix them immediately. all critical pages are available and behave as they used Rollbar Sentry No kind of testing can guarantee 100% error proof. We need to keep a balance between development effort and value. By using stateless end-to-end tests, we get a great value with minimal effort. In my experience, this approach works nicely for small and medium-size projects. Large-scale projects are usually invested heavily in monitoring, QA, and custom processes. Examples using Treo is a cloud infrastructure for that provides simple end-to-end testing flow and integrates with Slack & Github. Treo Lighthouse With Treo end-to-end testing setup is just a few clicks. To execute a test, just call an API and receive status update to Slack or Email in less than 20 seconds. A testing flow with Treo Personal website example: alekseykulikov.com Let’s look at a typical personal website: . It contains three pages: Home, Blog, and Not Found. https://alekseykulikov.com Typical personal website with three pages: Home, Blog, 404 Because I develop it locally and deploy with git push. These are three steps: build, upload to S3, invalidate CloudFront distribution. I want this site to be available, performant and accessible. I don’t want to validate deployment manually, by clicking and running Lighthouse audits for each page. Just . Why test this simple website? push and forget Step by step tutorial ( ): video Log in to using Github account (free trial is ~500 audits). https://treo.sh Click “Add new test suite” and name it; Add three pages with title and URL; Click “Create”, wait a few seconds, and look at results; Copy curl command to deploy script with an automatic message from Git. MESSAGE=`git log --format=%B -n 1 $sha1 | xargs`curl \-H "Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \-d message="$MESSAGE" https://api.treo.sh/v1/suites/1 Next time, I do changes and break accessibility on , Treo notifies me: /blog Failed Report Web application example: treo.sh Treo uses itself for testing. It is a SPA with 16 different pages. The testing process is similar to the first example with a few advancements. It uses cookies for authentication and special pages for performance monitoring in US/EU regions. Examples of UI treo.sh It’s quite natural to describe a SPA with a set of URLs. Usually, each URL is a router entry. Each entry is a separate view with custom logic. Loading the view without a difference from the previous run, gives a solid guarantee, that it continues to work. During development of Treo, I caught a lot of bugs using Treo. Broken pages, failed Webpack builds, accessibility and performance regressions. That gives me confidence about this approach. Stateless end-to-end tests are the most useful kind of tests for an invested amount of time. Next, go to unit tests, but this is . entirely different story I’m looking forward to hearing from the community! Let’s discuss an idea of stateless end-to-end tests on twitter or in comments. feedback @alekseykulikov_ Once again, it a universal approach. It is a set of trade-offs to achieve better quality and faster release cycle for small and medium-size projects. is not If you need any support with configuring Treo, contact me: . I’d be happy to help your team to define most important pages and look to Lighthouse results. info@treo.sh Let’s build reliable web apps. Happy testing! 👍