I’m Yusuke Shibui, I work for Launchable as a Senior Software Engineer. I live in the machine learning space and I’ve written a few books that discuss the productization of ML systems. Take a look if you may: Machine learning system design pattern (Japanese) Machine learning system development practical guide (Japanese) TL/DR: This blog discusses , which is a technique that applies ML to make test executions more efficient. If you are an engineer who struggles with test execution time/cost, read on. Predictive Test Selection Testing in today’s software development It was 2002 when the book “ ” came out, and since then, writing tests in software development have become the norm. Then, cloud and DevOps spread, thereby establishing the automatically to ensure code/product quality. There are all sorts of “tests” here: unit tests, integration tests, system tests, and so on. Test Driven Development practice of running tests in CI/CD pipelines Many repositories and projects adopted the rule that says pull requests can be only merged if adequate test cases are written and if all the tests pass. Nowadays, to add on top of this, it is common to see end-to-end tests which touch all the layers of the software from the end-user use case perspective. Among web services that include the UI/frontend pieces, e2e tests are table stakes, not just unit tests and integration tests. . It’s simply a must-have to introduce e2e tests to test such a product, covering user screen transitions Historically, CI/CD has been run on an engineer-managed environment like . Today, hosted standalone CI services are available, such as and . So are cloud services like and . E2E test execution management services include and . Writing code, including writing tests, is still dominated by local development computers, but increasingly test execution is shifting toward automation in free/paid cloud environments. Jenkins GitHub Actions CircleCI AWS Code Pipeline Azure Pipelines mabl autify As your services expand and code grows, tests also increase their volume/type in lockstep. Especially among long-running services, test cases increase like dead code increases: tests that you have no idea when they got in, tests that you don’t know what they are testing. : These problems have a detrimental effect on developer productivity for two reasons CI/CD turn around time & cost for unnecessary tests People cost and the loss of motivation to maintain unnecessary-looking tests As a result of running CI/CD on services, running tests costs money. Developers waiting for tests to finish adds to the time developers take to update/merge pull requests. Shorter cycle time leads to a rapidly improving valuable product getting in the hands of users. . As the product and the software grows, test execution time and developer efficiency tends to go down (or the cost goes up). To shorten test execution time, we often resort to parallelization of tests or test execution environment improvements, but that translates to more spend on CI/CD services, including time cost The second “people cost & motivation” factor is even more problematic. I’m sure every newly written test had a valid reason, but as the software changes, some tests inevitably become unnecessary. However, figuring out if it became unnecessary requires the knowledge of product/software specs. AFAIK, most teams take a more conservative approach to deleting tests, compared to deleting code – you don’t know who wrote those tests, and you don’t want to be held accountable to the quality problem caused by that removal. As a result, people who maintain those unnecessary tests end up spending time on efforts that they do not feel are valuable. My point is, writing tests and executing them in CI/CD aren’t enough to tame the challenges of maintaining quality and improving developer productivity. What is Predictive Test Selection? If the 2000s brought test-driven development, and the 2010s brought CI/CD, then one of the new trends in 2020s is Predictive Test Selection (PTS). Meta (formerly Facebook) presented this test execution efficiency technique, where machine learning is used to select tests that matter for a given pull request. Blog: Predictive test selection: A more efficient way to ensure the reliability of code changes Paper: https://arxiv.org/pdf/1810.05286.pdf The paper claims pass/fail of tests can be predicted by the code change and the past test execution records, to some extent. If you change code but not the relevant tests, that’ll likely increase the risk that those tests fail. If tests are failing often, regardless of the causes, the chance of this test failing next time is also likely high. Or if tests that are not modified for long time, and tests that have never failed, are unlikely to fail, setting aside for a moment whether those tests are valuable. When code is small and tests are a few, one might be able to infer “if you change this code, that test might fail”, and/or “these tests are failing often, maybe it’ll fail again.” But as the software gets larger, that becomes impractical. Predictive Test Selection uses code change log and test execution history as an input, and applies for a given pull request. machine learning to determine which tests are more likely to fail The paper uses the following features: Change history for files File cardinality Target cardinality Extensions of files Number of distinct authors Historical failure rate Project name Number of tests Minimal distance between files and targets Number of common tokens The target data of the training is binary test pass/fail. The above features are fed into gradient boosting decision tree (GBDT). GBDT is an algorithm that combines gradient boost, model ensemble, and decision trees. The popular implementation includes XGBoost and LightGBM. It’s known to be a very effective technique to learn from tabular data. The paper states that they chose GBDT because the algorithm is readily available, trains quickly, performs well when positive & negative cases are not balanced, and can process ordinal/categorical data. Test pass/fail data are recorded from CI/CD log per pull requests, along with code/test change data. Data for training gest accumulated automatically as CI/CD runs. Once trained, a model is used to select tests to execute (= tests that are likely to fail). Test results are recorded for evaluation and as future training data. . Executing successful tests results in unnecessary test execution wait time and test resources. Conversely, missing tests that fail results in potential software quality problems. Predictive Test Selection packs the selected tests with tests that are likely to fail as much as possible. By choosing a subset of meaningful tests from all the tests, it reduces the wait time, lowers the cost, and improves the quality by running tests that fail. By making changes go through tests, it makes the test execution cycle more efficient in software development. The value of the model is to find tests that are likely to fail Tests are reordered in the likelihood of failures. Imagine you have a test suite that takes 120 minutes to run. Let’s use Predictive Test Selection to select 60 minutes worth of tests that are more likely to fail. If you execute them randomly, on average engineers would wait for 30 minutes for a failure. Now if you execute tests in the order they are likely to fail, then the likelihood that this test suite will fail within the first 5 minutes get a lot higher. That means engineers can begin reworking the change after 5 minutes. This results in a faster time to CI/CD failure and shortens the time to start working on code fix. . In the age where CI/CD test execution is the norm and developers waiting for tests to come around is commonplace, I know you can see the value of CI/CD wait time going from 30 minutes to 5 minutes, regardless of your engineering org size Predictive Test Selection selecting tests to execute means tests that are not getting executed become clearer. Because it selects tests that are more likely to fail, based on code changes and test execution history, tests that are not getting selected consistently are tests that are unlikely to fail or tests that are unrelated to code that is changed (or tests that are covering infrequently changing code). . Put differently, these tests can be a prime target for removal, allowing test volume to come down (thus avoiding the maintenance overhead) Test data analysis like Predictive Test Selection reveals another challenge – flaky tests. Flaky tests are randomly failing tests. When you modify unrelated code, or when you are not modifying code at all, if a test fails for some reason (such as external reasons or limited resources) then those tests are said to be flaky. Flakiness is difficult to remove, and often simple re-execution is enough to make them pass, so tackling flakiness tends to be de-prioritized. What is a flaky test? Flaky tests are test cases that can pass and fail without changes to the code under test. By their nature, and lead to a loss of confidence in testing. flaky tests reduce developer productivity Predictive Test Selection infers tests that are likely to fail because of the code change at hand. So it’s desirable not to have flaky tests among the selected tests. Because flakiness is random, by recording test execution results, one can calculate the level of flakiness of a test. Predictive Test Selection can exclude unrelated flaky tests from the subset, thereby eliminating the unnecessary test execution and triaging of failures. (Note that it’s not good to ignore flaky tests; if you know the cause and it can be solved, it’s better to eliminate flakiness. In particular, flaky tests could still fail for legitimate reasons, and ignoring them could result in a software quality risk. See for more about flakiness.) this paper So there’s no wonder a practice like this emerged from Meta, which has a large software engineering team and provides planet-scale web services like Facebook, Instagram, Whatsapp, etc. Meta is not the only company that is doing this, too. Google published a paper “ ”. Making test execution efficient helps maintain software quality and improves the software developer productivity. Taming Google-Scale Continuous Testing Microsoft did “ “, where they used ML to assess the risk (test failures) of a commit. FastLane: Test Minimization for Rapidly Deployed Large-scale Online Services As the software gets larger and the engineering organization grows, automated test execution becomes a challenge that requires a creative solution. Though it was not Predictive Test Selection, Rakuten (Japanese) as well. deployed a solution to make test execution more efficient Test-driven development made writing tests table stakes; similarly, is running tests in CI/CD platforms such as Jenkins, GitHub Actions, and CircleCI. As the next step up from there, there’s a trend of making test execution efficient through techniques like . Predictive Test Selection . The translated version was also published . This post was translated from Japanese and originally appeared on Qiita here

The writer has included referral links in this story.

Using Machine Learning to Make Test Executions More Efficient

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Accelerate Your Pytest Performance for Enhanced Code Quality and Faster Feedback

Flaky Tests - A War that Never Ends

How to Deal With Flapping or Broken Tests with a new pytest plugin

How to Write Better Tests and Prevent Flaky Tests

Is Latency Slowing Down your E2E Tests?

Setting up a Perfectly Isolated End-To-End Environment

Accelerate Your Pytest Performance for Enhanced Code Quality and Faster Feedback

Flaky Tests - A War that Never Ends

How to Deal With Flapping or Broken Tests with a new pytest plugin

How to Write Better Tests and Prevent Flaky Tests

Is Latency Slowing Down your E2E Tests?

Setting up a Perfectly Isolated End-To-End Environment

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps