It’s (been) Time to Use Machine Learning to Fix End to End Testing

By Izzi Azeri, mabl co-founder

Disclosure: mabl, the ML-driven test automation service, has previously sponsored Hacker Noon.

I’m a serial entrepreneur who’s spent time working at Google, VMware, Stackdriver and now my second startup I co-founded, mabl. We started mabl earlier this year to help developers with the painful challenge of testing their applications, especially as development cycles are getting shorter with the adoption of CI/CD and continued automation. As we started the company, I did a lot of research, primarily by reaching out to developers to understand their take on software testing, especially those that have embraced DevOps and we also launched a survey to get quantitative feedback. In addition, I also found lots of blogs written about the challenges of testing applications in a modern developer workflow. One of the blog posts that I was drawn to was from Mike Wacker, a software engineer at Google who was also a developer at Microsoft previously. Mike wrote a blog post in 2015 about why End to End Tests just don’t matter anymore and developers should focus on Unit Tests primarily, and to a lesser extent, Integration Tests, as the basis for ensuring quality in their code. If you haven’t read it and are involved in testing, you should. His premise was (I’m summarizing and probably not doing enough justice to Mike):

• E2E testing “should” deliver the most value for customers, since they’re replicating real user scenarios. Mike argues that all product stakeholders including developers, QA, manager’s agree to this general premise.
• E2E tests however are not reliable (=flaky), take too long to complete (=running overnight), and when there are bugs, the cause is hard to diagnose. Mike argues that these issues with E2E tests should force us to consider a different testing approach for ensuring customers are happy.
• Unit tests are the immediate solution because they find bugs and remedies faster. These, combined with Integration tests which help test numerous components together, identifying bugs before lengthy E2E tests are written, are the solution recommended by Mike to the broken promises of E2E tests.

Mike was totally correct with his conclusion, for the time. Why waste time on E2E tests when you already know the challenges they present? Instead, compensate by adapting your testing process.

Enter Machine Learning

As Sundar Pichai discussed at IO 2017, Google is revamping all of its products to be AI first — given the advancements in computer vision, voice recognition, and machine learning. We’ve already seen the advancements Google has delivered from a consumer perspective for years, from self-learning thermostats, to traffic navigation, to speech recognition. The next step for Google is applying AI for business use cases, which it’s already doing within Google Cloud. At mabl, we agree with Google. Specifically, we believe ML could have a profound impact on improving the lives of developers, allowing them to spend more time building great products and less time on repetitive tasks. This is why we’re challenging ourselves to solve the specific problems that Mike pointed out with E2E testing.

Some of the things our team is focusing on include:

• 
  

  Flaky tests-as Mike mentioned in his blog, tests often fail due to the test not being reliable. When this happens, developers lose confidence in their tests and just ignore them. We believe mabl should be able to automatically detect when steps in trained tests are failing because of a slight change to the UI. Mabl should recommend a fix to the user and automatically update the test script as well. Our goal is to eliminate flaky tests so developers can spend more time on coding and less time on maintaining their tests.

• 
  

  Isolating failures-A developer searching for a needle in a haystack to solve a test failure isn’t the best use of that developers time. Given that mabl has different types of machine-created tests running all the time, she’s learning much about the application and can use use this data to provide evidence of what and where pieces of an application aren’t working. Helping developers figure out what is wrong and why is of high importance for mabl.

• 
  

  Speed of tests-Mike specifically called out waiting for tests to run overnight, just to find out if a specific component is broken. We’ve also heard from several of our users that manual QA or even outsourced QA to manual testers still takes the time of a normal human to test. We believe tests should be continuous, not overnight, and should run at the speed of machine time, not human time.

• 
  

  Self-learning tests-One thing Mike didn’t mention is about learning from past test runs. We believe mabl should continuously learn from the tests she’s running as well as the feedback users are giving her, just like ML applied to consumer patterns. The more she learns, the smarter mabl can be about Insights she’s providing users about the quality of their applications.

mabl is solving hard problems

As Mike pointed out, end to end testing is difficult today (and @2015 when Mike wrote his blog post). However instead of replacing E2E with Unit and Integration test coverage, we believe we should fix E2E testing. We believe advancements in machine learning allow us to both fix E2E testing and even go beyond what a human is able to do with test automation frameworks. We’ve been at this for 8 months and we’ve learned a lot along the way. We’ve solved some hard problems but still have many more to tackle. If you want learn more, reserve your spot for early access as we’re launching the service very soon.

Originally published at www.mabl.com on October 25, 2017.