mabl Uses AI to Bring Software Testing into the DevOps Era

Founder Interview

Disclosure: mabl, the ML-driven test automation service, has previously sponsored Hacker Noon.

Today, we’re going to catch up on the state of this startup — and find out what makes Co-Founder & CEO Dan Belcher do what he does. Also read the other half of this interview, The Entrepreneurial Journey of mabl Co-Founder Dan Belcher.

David Smooke: Why must your company exist now?

Dan Belcher: mabl exists now because the pace of software development has increased dramatically over the last ten years, to the point where many teams can develop meaningful changes every day, but QA can’t keep up, so you either have to slow down or sacrifice testing and quality. Our view is that the state of QA is holding back innovation across the industry, and mabl is here to fix that.

David: So you started in January 2017. What did you achieve this year?

Dan Belcher: Oh it’s been a very exciting year. My co-founder Izzy and I left Google at the end of January. We started building the team in February, and we started writing code in April. The most important achievement this year has been building a world class team. We started with engineering, which is now 12 people and growing, and now we’re turning our attention to marketing. We were lucky to hire Mike, here in the room with me, as our first marketing lead, a couple months ago, and that team will grow pretty quickly.

“Drive” is one of our core values as a company, and I think that comes across in the pace. We’ve been working on the product for about 10 months now, it’s being used extensively by many companies, and we’re getting great feedback on the alpha. mabl is testing hundreds of applications every day, and we’re a team that’s approaching 20 people here in Boston, and we’re excited to round the corner into 2018.

“Drive” is one of our core values as a company, and I think that comes across in the pace.”

David: What do you guys hope to achieve in 2018 in terms of company, traction and product?

Dan Belcher: At a high level we feel really great about the team and will continue to grow the team incrementally in 2018. The product is now at a point where we’re getting great feedback, so you’ll see us open up a public BETA soon, making it so anyone can get access to mabl. And then we’ll launch the service generally and scale from there. Importantly, we also have some more difficult problems that we’ve been working on that require long-term R&D that we’ll finally get in users’ hands this year, so we’re really excited about that.

David: With mabl, she runs functional tests on your app or website and then the more she learns, the more she does it. So could you explain a little bit about her learning logic?

Dan Belcher: In terms of test output, one of the big problems in this space is that it’s hard to understand when the product is broken, versus when a test is broken. That’s really challenging because the output of most testing tools is raw logs and you have to wade through all of this verbose technical data. So, one thing mabl is really good at is processing all of the test output and learning what normal looks like for this application, and when something is out of the ordinary. We do that by constructing machine learning models from the test output and predicting how an app should behave, observing when something isn’t doing what we predicted.

“One of the big problems in this space is that it’s hard to understand when the product is broken, versus when a test is broken.”

The more tests we run, the more confident mabl can be in terms of expected behavior, so when we see something’s different, we can react to it. One of the things mabl will notice is if a page loads much slower in one execution of a test than it has historically. Perhaps you’ve changed a query that inadvertently increases the load time for that page. This type of thing can go unnoticed by QA engineers and other testing tools, but since mabl has been observing the normal variation in load times for the page every time we run a test, she becomes increasingly confident in her predictions.

Another example is visual changes. mabl takes screenshots at every step in a test. Over time, she’s not just able to identify when part of a change pages but she can also predict what changes might be meaningful by learning from historical changes.

David: With technology and new products, a lot of times there’s a fine line between eliminating jobs versus empowering people that do the job. How does mabl save a company’s money and how does it make a company money?

Dan Belcher: Yeah, great question. Most of the customers that we’re working with are really driven by speed and innovation. What they want is to deliver high quality product to their customers very rapidly. mabl is less about saving you costs, and more about making it so that you can innovate and iterate faster. The way that actually ends up playing out is QA can offload the burden of a lot of routine end-to-end, functional testing, and then they get to be more proactive, focusing on defining more and better test scenarios, performing root cause analysis, working on security, usability, etc.

To illustrate, I’ve certainly been on multiple software teams where we say we really want to be as responsive (to mobile users) as we can, we never get around to testing for responsiveness because we’re just trying to keep up with development on our core clients (such as Chrome on desktop). If test automation had been handled by a service like mabl, we would have had time to invest more in mobile support.

David: You refer to mabl as “she” across the website and the product which is uncommon for the tech industry where many of the brand personifications are masculine. Could you share a little bit about this choice and also where the name for the company originated?

Dan Belcher: We wanted mabl to be a persona because we wanted people to feel like they were inviting another member onto their team, and so we had to give it a name. One thing we like about the mabl brand is that it’s disarming and friendly. It’s not a brand that claims to be an all knowing sort of God-like thing in the background or Wizard of Oz, or something like that. We wanted mabl to be very approachable and helpful… and sort of fun. Those were the attributes we went after when we were defining the brand and coming up with the name. And of course, it’s really easy to say mabl and easy to spell if you can remember to drop the ‘e’ at the end.

“We wanted people to feel like they were inviting another member onto their team, and so we had to give it a name.”

We’re concerned about the tendency for people to brand AI-based tools and people to interpret AI-based brands as being very complex and advanced. We don’t think of ourselves as a ML or an AI company, we think of ourselves as a testing company that just happens to use a lot of very complicated technology in the background, and we think the mabl brand sort of reinforces that. The idea of mabl is really simple; it’s just a service that can learn how software works, write some tests, run the tests, let you know when something breaks. It’s very difficult from an implementation perspective, but that’s not the users’ concern.

David: Yeah, I like the customer-focused approach because I do think too many brands, use just a little bit of AI or Machine Learning and across the top of their homepage it’s like AI for this or Machine Learning for that, and it’s like that really has nothing to do with the customer’s need and problem… So, what are mabl’s biggest threats?

Dan Belcher: We see two primary threats. One problem with QA not having enough time is so acute right now is that we don’t have time to invest in better ways of doing the work. Some companies that we’ve spoken with have sort of given up. They say alright, look we tried doing test automation, we invested a lot and the tests are very flaky and brittle, and every time we change our product all the tests break and so what we’ve decided is to change our strategy to not do very much functional or end-to-end testing and let our customers find the bugs’ I hope we can show people that there is a better way.

The other risk is very related. This is not an easy problem from a technology perspective that we’re trying to solve, and it’s never been done before. We’re applying new machine learning models to QA, combining them with other analysis tools, doing that at scale, in real-time, etc. And so that risk is on us and our ability to innovate; can we solve that problem in a general way that thousands of customers find useful?

David: With your early customers could you speak to where they’re finding the greatest value and how you’re changing their actual software development?

Dan Belcher: When you sign up, you point mabl to a test environment for one of your applications and give her some credentials, as you would if you hired a new QA person named mabl. You’d say ‘alright go check out the app and kind of get yourself familiar with it, In order to accomplish that mabl will basically figure out how to login into the app and then crawl. So we’ll visit every page in the app, we’ll crawl, look at your web app, your docs, your marketing site, and so forth. The reason we built that feature, was to help train these models around how the application works. We also see what’s normal as we do this, so we’re able to surface really useful insights to the user when things aren’t normal.

For example, we find every one of your broken links across your entire application, website, docs, and so forth. We find every page that has a JavaScript error that loads when we run a test. We monitor page load times, and let customers know when the timing of a page load changes, or that running a test changes. We’ll find visuals diffs on pages.

“We find every one of your broken links across your entire application, website, docs, and so forth.”

So that crawling feature that we actually thought was a fairly small part of the customer value, turns out to also be very important to user because they don’t have time to go out and look for every JavaScript error or every broken link, and so forth. And so within a few minutes of signing up for mabl they’re getting those insights that otherwise they wouldn’t see.

David: That sounds like a pretty strong ah-ha moment for them, it’s pretty cool to get that value so quickly.

Dan Belcher: There are customers whose apps have 50,000 or more pages, and we’ll find which of those 50,000 pages have new JavaScript errors that were introduced by a recent release. You can just imagine how long it would take a QA team, even a large QA team, to be able to make those discoveries for those types of issues. With mabl that’s a matter of hours for an application at that scale.

Read the other half of this interview, The Entrepreneurial Journey of mabl Co-Founder Dan Belcher.