“Autify" is a AI-powered no-code software test automation platform that enables anyone to easily automate software testing, the process of verifying that developed software works as expected

Autify

autify

I’m Yusuke Shibui, I work for Launchable as a Senior Software Engineer. I live in the machine learning space and I’ve written a few books that discuss the productization of ML systems. Take a look if you may: Machine learning system design pattern (Japanese) Machine learning system development practical guide (Japanese) TL/DR: This blog discusses , which is a technique that applies ML to make test executions more efficient. If you are an engineer who struggles with test execution time/cost, read on. Predictive Test Selection Testing in today’s software development It was 2002 when the book “ ” came out, and since then, writing tests in software development have become the norm. Then, cloud and DevOps spread, thereby establishing the automatically to ensure code/product quality. There are all sorts of “tests” here: unit tests, integration tests, system tests, and so on. Test Driven Development practice of running tests in CI/CD pipelines Many repositories and projects adopted the rule that says pull requests can be only merged if adequate test cases are written and if all the tests pass. Nowadays, to add on top of this, it is common to see end-to-end tests which touch all the layers of the software from the end-user use case perspective. Among web services that include the UI/frontend pieces, e2e tests are table stakes, not just unit tests and integration tests. . It’s simply a must-have to introduce e2e tests to test such a product, covering user screen transitions Historically, CI/CD has been run on an engineer-managed environment like . Today, hosted standalone CI services are available, such as and . So are cloud services like and . E2E test execution management services include and . Writing code, including writing tests, is still dominated by local development computers, but increasingly test execution is shifting toward automation in free/paid cloud environments. Jenkins GitHub Actions CircleCI AWS Code Pipeline Azure Pipelines mabl autify As your services expand and code grows, tests also increase their volume/type in lockstep. Especially among long-running services, test cases increase like dead code increases: tests that you have no idea when they got in, tests that you don’t know what they are testing. : These problems have a detrimental effect on developer productivity for two reasons CI/CD turn around time & cost for unnecessary tests People cost and the loss of motivation to maintain unnecessary-looking tests As a result of running CI/CD on services, running tests costs money. Developers waiting for tests to finish adds to the time developers take to update/merge pull requests. Shorter cycle time leads to a rapidly improving valuable product getting in the hands of users. . As the product and the software grows, test execution time and developer efficiency tends to go down (or the cost goes up). To shorten test execution time, we often resort to parallelization of tests or test execution environment improvements, but that translates to more spend on CI/CD services, including time cost The second “people cost & motivation” factor is even more problematic. I’m sure every newly written test had a valid reason, but as the software changes, some tests inevitably become unnecessary. However, figuring out if it became unnecessary requires the knowledge of product/software specs. AFAIK, most teams take a more conservative approach to deleting tests, compared to deleting code – you don’t know who wrote those tests, and you don’t want to be held accountable to the quality problem caused by that removal. As a result, people who maintain those unnecessary tests end up spending time on efforts that they do not feel are valuable. My point is, writing tests and executing them in CI/CD aren’t enough to tame the challenges of maintaining quality and improving developer productivity. What is Predictive Test Selection? If the 2000s brought test-driven development, and the 2010s brought CI/CD, then one of the new trends in 2020s is Predictive Test Selection (PTS). Meta (formerly Facebook) presented this test execution efficiency technique, where machine learning is used to select tests that matter for a given pull request. Blog: Predictive test selection: A more efficient way to ensure the reliability of code changes Paper: https://arxiv.org/pdf/1810.05286.pdf The paper claims pass/fail of tests can be predicted by the code change and the past test execution records, to some extent. If you change code but not the relevant tests, that’ll likely increase the risk that those tests fail. If tests are failing often, regardless of the causes, the chance of this test failing next time is also likely high. Or if tests that are not modified for long time, and tests that have never failed, are unlikely to fail, setting aside for a moment whether those tests are valuable. When code is small and tests are a few, one might be able to infer “if you change this code, that test might fail”, and/or “these tests are failing often, maybe it’ll fail again.” But as the software gets larger, that becomes impractical. Predictive Test Selection uses code change log and test execution history as an input, and applies for a given pull request. machine learning to determine which tests are more likely to fail The paper uses the following features: Change history for files File cardinality Target cardinality Extensions of files Number of distinct authors Historical failure rate Project name Number of tests Minimal distance between files and targets Number of common tokens The target data of the training is binary test pass/fail. The above features are fed into gradient boosting decision tree (GBDT). GBDT is an algorithm that combines gradient boost, model ensemble, and decision trees. The popular implementation includes XGBoost and LightGBM. It’s known to be a very effective technique to learn from tabular data. The paper states that they chose GBDT because the algorithm is readily available, trains quickly, performs well when positive & negative cases are not balanced, and can process ordinal/categorical data. Test pass/fail data are recorded from CI/CD log per pull requests, along with code/test change data. Data for training gest accumulated automatically as CI/CD runs. Once trained, a model is used to select tests to execute (= tests that are likely to fail). Test results are recorded for evaluation and as future training data. . Executing successful tests results in unnecessary test execution wait time and test resources. Conversely, missing tests that fail results in potential software quality problems. Predictive Test Selection packs the selected tests with tests that are likely to fail as much as possible. By choosing a subset of meaningful tests from all the tests, it reduces the wait time, lowers the cost, and improves the quality by running tests that fail. By making changes go through tests, it makes the test execution cycle more efficient in software development. The value of the model is to find tests that are likely to fail Tests are reordered in the likelihood of failures. Imagine you have a test suite that takes 120 minutes to run. Let’s use Predictive Test Selection to select 60 minutes worth of tests that are more likely to fail. If you execute them randomly, on average engineers would wait for 30 minutes for a failure. Now if you execute tests in the order they are likely to fail, then the likelihood that this test suite will fail within the first 5 minutes get a lot higher. That means engineers can begin reworking the change after 5 minutes. This results in a faster time to CI/CD failure and shortens the time to start working on code fix. . In the age where CI/CD test execution is the norm and developers waiting for tests to come around is commonplace, I know you can see the value of CI/CD wait time going from 30 minutes to 5 minutes, regardless of your engineering org size Predictive Test Selection selecting tests to execute means tests that are not getting executed become clearer. Because it selects tests that are more likely to fail, based on code changes and test execution history, tests that are not getting selected consistently are tests that are unlikely to fail or tests that are unrelated to code that is changed (or tests that are covering infrequently changing code). . Put differently, these tests can be a prime target for removal, allowing test volume to come down (thus avoiding the maintenance overhead) Test data analysis like Predictive Test Selection reveals another challenge – flaky tests. Flaky tests are randomly failing tests. When you modify unrelated code, or when you are not modifying code at all, if a test fails for some reason (such as external reasons or limited resources) then those tests are said to be flaky. Flakiness is difficult to remove, and often simple re-execution is enough to make them pass, so tackling flakiness tends to be de-prioritized. What is a flaky test? Flaky tests are test cases that can pass and fail without changes to the code under test. By their nature, and lead to a loss of confidence in testing. flaky tests reduce developer productivity Predictive Test Selection infers tests that are likely to fail because of the code change at hand. So it’s desirable not to have flaky tests among the selected tests. Because flakiness is random, by recording test execution results, one can calculate the level of flakiness of a test. Predictive Test Selection can exclude unrelated flaky tests from the subset, thereby eliminating the unnecessary test execution and triaging of failures. (Note that it’s not good to ignore flaky tests; if you know the cause and it can be solved, it’s better to eliminate flakiness. In particular, flaky tests could still fail for legitimate reasons, and ignoring them could result in a software quality risk. See for more about flakiness.) this paper So there’s no wonder a practice like this emerged from Meta, which has a large software engineering team and provides planet-scale web services like Facebook, Instagram, Whatsapp, etc. Meta is not the only company that is doing this, too. Google published a paper “ ”. Making test execution efficient helps maintain software quality and improves the software developer productivity. Taming Google-Scale Continuous Testing Microsoft did “ “, where they used ML to assess the risk (test failures) of a commit. FastLane: Test Minimization for Rapidly Deployed Large-scale Online Services As the software gets larger and the engineering organization grows, automated test execution becomes a challenge that requires a creative solution. Though it was not Predictive Test Selection, Rakuten (Japanese) as well. deployed a solution to make test execution more efficient Test-driven development made writing tests table stakes; similarly, is running tests in CI/CD platforms such as Jenkins, GitHub Actions, and CircleCI. As the next step up from there, there’s a trend of making test execution efficient through techniques like . Predictive Test Selection . The translated version was also published . This post was translated from Japanese and originally appeared on Qiita here

Using Machine Learning to Make Test Executions More Efficient

In this article, I will share a field, specifically with the . I want to help you select the best tool for your context, so I’m offering a centralized place with information about the different options on the market. state of the art in the test automation low code approach It’s kind of difficult to distinguish if this is another marketing-created buzzword in the software development world. In literature (blogs, websites, tech journals), I’ve observed how different terms are used for the same idea, including “low code”, “no-code”, “codeless”, “scriptless”, and I’m probably missing some. Basically, “low code” refers to an alternative approach to traditional software development. Instead of defining the expected behavior of the system in code, it tends to be more visual and uses little or no code at all. As I see it, we are increasing the level of abstraction of the language we use to express to machines what to do. We started programming circuits, strings with 0’s and 1’s, assembler instructions, and then different languages, providing more and more abstraction layers. Each layer is removing complexity, closing the gap between the human and the machine. The flip side is that the abstraction also has a cost, typically associated with performance or flexibility, but it typically makes sense to pay that cost in terms of productivity and results. We could say that low code is another abstraction layer on the top, and we are also seeing how we can benefit from this approach for test automation. Low code platforms for test automation aim to simplify the test automation with functionality that doesn’t require the user to write almost any code. You will have a recorder that will let you create test cases easily and edit them with a simple interface, without requiring coding skills. In the last few years, new low code solutions for test automation have been spinning up. There is an interesting cycle: more entrepreneurs are trying to build this type of tool, more investors are supporting them, and the testing community adopts the tools and provides feedback. These are just a few signs that it’s an interesting approach for test automation to follow. Today, we are releasing the first draft of our research, just with the list of tools, as another step toward discovering blind spots and getting early feedback. The next steps are: Describe the details of our research method. Add more information about each tool, highlighting the main characteristics. Analyze results and give our perspective and conclusions. Some considerations: We included only test automation tools for web and mobile. For now, we excluded tools that are specific to a certain platform such as Salesforce or GeneXus (that’s why you won’t find or in the list). Provar GXtest We are trying to exclude those tools that don’t provide enough information to evaluate them and will continue filtering the list as we progress through the research. Low Code Tools for Test Automation ACCELQ AI Codeless Testing Help Center AscentialTest Autify Avo Automation Cerberus Testing CloudQA Co.meta Endtest Functionize Ghost Inspector Ginger iNSta IntelliSWAUT Katalon Kobiton Leapwork Mabl MuukTest Perfecto Scriptlessl Preflight QARA Enterprise Rainforest QA Rapise Cross Browser Testing Reflect.run Sahi Pro Sofy.ai Stela test Telerik test.ai TestArchitect TestComplete Testim TestingWhiz TestProject testRigor Testsigma Usetrace Virtuoso QA Waldo Worksoft ZAPTEST ZeuZ Asking for Feedback If you know a test automation tool with a low-code approach that is not included in the list according to the criteria we followed, please let us know and we will review it, add it to the list if it corresponds, and we will also try to understand why we didn’t find it in our search. I want to thank Danny Gutiérrez, who performed most of the research, also with the help of Andrea González, Leticia Martínez, and Lucía González. Follow us on , , and , to be part of our community! LinkedIn Facebook Twitter Also published here

Low Code Tools for Test Automation

A 12-year-old coder from north London has made headlines around the world selling nearly $400,000 worth of programmable NFT artwork. The success of Benyamin Ahmed has now won positive press from major outlets including , , , , , , , , and . The New York Times Business Insider CNBC The Telegraph BBC ITV News Mirror NY Post Bitcoin.com Decrypt The visual style of ’s artworks - a total of 3,350 ‘Weird Whales’ - shares similarities to Crypto Punks, but switches punk characters for kawaii crypto whales. The project began as something to keep the whizz kid occupied during his Summer holidays, but a simple and compelling idea, and the buoyant nature of the market, helped the boy turn a 6-figure profit. Benyamin Now the boy genius, who has never owned a bank account, is sitting on a sizable Ethereum nest egg for his future. Better still, trading of the whales on the secondary market provides the boy with steady income, as a small percentage from each sale is diverted into Benyamin’s Ethereum crypto wallet. The success of Benyamin is another bullish indicator for the NFT market, and provides a wake up call for anyone who hasn’t yet realised what industry insiders already know: there has never been a better time to create, sell or buy NFTs. Weird Whales by Benyamin Ahmed Smashing Records Again The $400,000 which collectors spent on Benyamin’s pixelated cetaceans is a relative drop in the NFT ocean. According to , the Crypto Punks collection of NFTs is now worth over $1 billion. A number of other projects are also highly sought after and much in demand. The collection is currently worth $697 million, and is not too far behind at $433 million. data from CoinMarketCap (CMC) ArtBlocks Bored Ape Yacht Club NFT Trading is also incredibly healthy, with seven day sales figures for the market at . Chart data on NonFungible.com shows that the market is smashing records left and right, surpassing the previous high water mark for the industry which was back in May. NFTs are now big business and the potential rewards for projects which can capture the imagination of investors is massive. $5.8 billion Primary sales of NFTs smashed records in August (Source: NonFungible.com ) Multiple Markets & Marketplaces The health and dynamism of the NFT market is reflected in the marketplaces which have sprung up to facilitate the trade of NFTs. Some of the names which have already established themselves in the industry include , and . OpenSea Rarible SuperRare Big name players such as Binance have also entered the NFT market, launching not one, but two NFT marketplaces. is a centralized NFT market hosted on Binance.com, while is a decentralized, non-custodial NFT market hosted on Binance Smart Chain. When Binance moves in hard on any one sector in the market, it’s time for everyone else to sit up and take notice. Binance NFT Featured by Binance At the same time there are small, emerging marketplaces that are seeking to distinguish themselves by focusing on a specific niche, such as , which bills itself as a ‘green’ music NFT platform. OneOf At the other end of the market has a different philosophy entirely, as it focuses on bringing the many disparate NFT niches together under one roof. The project is building a community hub that supports art, music and gaming NFTs. In this way Artify can become a one stop shop for the entire industry. It’s a strategy which could prove to be wise in the long run, as the NFT market continues to grow far beyond art and collectibles. Artify A Supportive Community While the NFT sector may be generating billions of dollars and growing apace, it still retains a sense of camaraderie and community among its members. When Benyamin Ahmed was creating his Weird Whale collection he found there was no shortage of people willing to help him achieve his goals. One of the companies which assisted him the most was . In a remarkable display of kindness the company gifted Benyamin the scripts they used to create their own programmable NFT collection. Try to think of another billion dollar industry in which the major players freely give away their intellectual property to 12-year-olds. Boring Bananas now offers their full GitHub repository through their , helping anyone get a leg up into the industry. Boring Bananas Discord Beyond Art Art may have been at the vanguard of the NFT market for some time now, but there are strong signals that as the market matures, other use cases are emerging and increasing in importance. , a Pokemon-style gaming project, now supports an NFT collection worth $1.2 billion. That places it ahead even of Crypto Punks in terms of value. Axie Infinity Everywhere there are positive signs for the NFT market and lots of reasons to get involved. With a strong sense of community, a great deal of fun to be had, a range of marketplaces to suit all tastes, and the chance to make considerable profits, the NFT market has a lot to offer. We’re still in the early days of this industry, but there’s never been a better time to get involved than now.

Stories, News & Mentions about AUTIFY

Autify’s Stories on HackerNoon