We hear the term ‘continuous testing’ a lot in the software testing industry. It’s one of those buzzwords that just keeps coming up, time and time again.
But what does it actually mean? And why is it helpful?
Previously, continuous testing was simply seen as the process of executing automated tests as part of the software delivery pipeline.
But, in fact, continuous testing is now being understood as much more than that.
When writing Leading Quality, I spoke to QA experts and thought leaders who shared their thoughts on testing best practices. These conversations painted a clearer picture of how top organisations structure their QA strategy and the techniques they use to achieve quality.
It was clear to me that continuous testing was top of mind for many of these individuals.
In this article, I’m going to be discussing the key differences between continuous testing and traditional testing, and the schools of thought in the QA community regarding the benefits of each method. It is my belief that testing continuously can have many interesting benefits, and implementing it effectively can help you achieve quality results.
Continuous testing does what it says on the tin - it’s all about testing continuously. This means testing software, apps or websites at every stage of their lifecycle. If you are truly executing continuous testing, you are incorporating testing even before a line of code is written, and even beyond release.
This means you are pre-empting problems before they happen - putting quality at the forefront early on. Skills you might use to test an application in traditional testing are being used to test a concept, idea, or the very early stage designs that are not fully formed yet. This is similar to the mentality behind TDD (Test-Driven-Development).
Essentially, you are building testability into your product, and code, right from the very start.
Dan Ashby wrote an extremely insightful article on this topic, including a fantastic graph to demonstrate what it means to truly test continuously.
Basically it means… you are always testing! But the testing can, in fact, begin even before the plan. Dan says:
“Testing the idea is vital; investigating to uncover information, refactoring the idea to enhance and solidify it.”
What Dan is suggesting is that the idea of ‘testing’ doesn’t necessarily mean writing an automated test script, or something similar, and sending it off. You can test an idea, discussing its potential faults and bugs, and iron out issues before embarking on a formal, traditional testing process.
Traditionally, QA and testing had both been considered an afterthought - running a series of tests right before release, at the end of the product lifecycle.
What this traditional method means is that the product owner thinks of an idea, decides whether it is viable, sends it off to the dev team to code, who then send it off to the QA team for testing before release.
This sounds simple enough. But in fact, this means the software development process becomes too linear. Rather than a cyclical process, where QA is involved from the very beginning, this traditional method passes the product down the line prioritising speed over quality - and it does mean that bugs appear along the way.
But, quality can’t suffer in favour of speed of release.
That’s where continuous testing comes in.
This might sound like a lot more work. And for teams strapped for time and under a lot of pressure to release fast, extra work could seem near impossible. The idea of extra testing at every stage of a product lifecycle could concern even the slickest engineering team going. But continuous testing doesn’t need to slow down your release process, or require extra, expensive internal hires.
You can implement continuous testing without it becoming a bottleneck.
So how do you implement this technique effectively?
I find research conducted by Capers Jones useful when expressing why this method works.
Jones discovered that, in a typical release cycle, a bug found post-release could cost $16,000 or more to address. That’s enough to make any CTOs eyes water - especially when budgets are strapped.
But, what Jones found is that developers could have fixed that same bug in the early stages for as little as $25. Just take a look at the graph he created below:
What this graph shows is that the highest percentage of defects are introduced in the early stages of the development process. These are the cheaper bugs to fix - so why don’t we fix them earlier on? Why not save precious time, and money, but testing before it's too late?
Continuous testing allows you to catch these frustrating bugs before they become bigger problems. It’s so crucial to think about quality early on, to save you money, unnecessary work, and bugs that cause your team a lot of hassle.
When done correctly, continuous testing ensures quality in the software you create, reduces critical bugs and saves you money.
Keeping QA front of mind decreases the likelihood of a huge, expensive bug being discovered right before release date and putting the brakes on the process.
In this way, more testing is not a bottleneck, but in fact, a way to build the foundations for successful future releases.
Quality is crucial. And your customers will always expect it. By testing early, and often, you can improve the quality of your software, and find the critical bugs before your customers do.
Author Bio: After selling his first startup, Ronald Cummings-John is now scaling up Global App Testing, a VC-backed, crowdsourced testing platform with over 40,000+ professional testers globally. Global App Testing was selected as one of the fastest-growing tech companies in the UK. His passion for quality assurance has taken him around the world, and he has worked with QA and product teams from top companies including Etsy, Microsoft, King, Spotify, and eBay. Ronald is co-author of the book, "Leading Quality: How Great Leaders Deliver High-Quality Software and Accelerate Growth.