The Early Velocity Trap

With much of the DevOps stack having matured and good practices becoming more mainstream, the need for quality in software has quickly become a central issue. This article is the first in a series in which we’ll shine the spotlight on software quality from a DevOps context.

Introduction

Trends have companies focused on customer experience as a way to differentiate themselves from their competitors. However, it seems that one of the worst-kept secrets in the software industry that quality is still one of the most reliable ways to differentiate your company from your competitors. If your product is constantly breaking and your competitors are more reliable and have comparable functionality, your organization will eventually run out of customer goodwill.

Also, there are high costs that stem from poor quality. Maintenance takes longer, and that stifles innovation. Toilsome work leads to burnout, which leads to attrition. That, in turn, leads to lost organizational knowledge, which exacerbates maintenance, onboarding, and innovation. So it’s no surprise that one of the few things that business people, managers, and software engineers agree on is that quality is of critical importance.

Curiously, when it comes time to take the actual steps to build quality in, it’s often the first thing to go. It often becomes a choice between delivery “on time” or quality. The irony here is that, like so many aspects of software, focus on one concern is often counter-productive and ends up being harmful to our original intent. In this case, the habit of prioritizing delivery over quality usually leads to longer and longer delivery times.

So how might we change the trajectory of software organizations struggling with this problem? This behaviour isn’t something that can be talked-away or an issue that will magically disappear once some fancy new technology is in place. Cultures that generate technical debt generally need to change their approach on some fundamental level.

In this article, we’ll examine one of the most typical pitfalls around software quality and look at ways to resolve the issues that arise from it.

The Early Velocity Trap

It feels great when you’re working on new features. Every sprint, you’re wowing your stakeholders by showing off snazzy new functionality (this is especially prevalent at the beginning of a project when everything is greenfield). Everyone is happy and the best of friends because there’s a clear feeling of progress and innovation, and it’s fun. Your team is developing with this incredible velocity for weeks or even months — you’re bonafide heroes.

Unfortunately, there’s a problem lurking under the surface that entire time. It is the most insidious problem in software. Developers don’t notice it because it’s hard to see that things get a little slower every week. Stakeholders don’t notice it because it makes sense that delivery plateaus. After all, now the team is working on the “hard stuff.”

Where this begins to hit home is the development workflow. Builds that took seconds now take minutes. Unit tests (if you have them) take minutes to run, and forget about those integration tests — now they’re up to more than an hour! Code coverage drops, dependency versions are out of date, SonarQube issues seem to be breeding like jack-rabbits, and architectural changes that we’ve been putting off now seem like significant undertakings (8 points at least).

In response, the team agrees to throw a “technical debt user story” into their backlog and to “get to it as soon as we have time.” Or even worse, the team says, “we’ll get to it later,” and proceeds to forget about it.

Deferring the problem is the technical debt trap.

https://knowyourmeme.com/memes/its-a-trap

Before we explore a solution, let’s understand the consequences of simply not addressing tech debt. This approach is an option that many people try to rationalize. Examples:

“This change isn’t going to live past three months, so we don’t need unit tests.”

“There’s a new major version that’s coming out later this year, and we’ll have to do this again then. Do we want to do this twice?”

“Nobody is going to reuse this”

“The Customer isn’t willing to pay for quality — they want features!”

If left long enough, the decision to ignore tech debt inevitably leads to one place: the end of the product. That could mean many things, such as the end of the team, cancellation of the product, putting it into maintenance mode, divestiture, or even the company's failure.

Any ignored technical debt starts a downward spiral that leaves teams struggling to stay afloat as they swim in a slowly increasing pool of defects. Eventually, forward progress halts because defects have grown beyond the ability of the team to fix anything. As a result, life becomes misery because the brutal workflows are laborious for developers to work in, and stakeholders eventually get impatient and demand changes. These conditions can lead to rushed features where changes get rammed through, resulting in even more misery.

To make matters worse, as DevOps practices have gained traction, elite performers have widened the gap between competitors slower to the game. In the old days, the downward spiral was a slow one — teams could make significant progress for months or even years with minimal effort put towards quality. Nowadays, that downward spiral happens a lot faster if you make the mistake of accelerating delivery (say, with a Continuous Delivery pipeline) without having quality-promoting practices in place.

Solution: The Three Ways of DevOps

If any of this sounds familiar, please know that it doesn’t have to be like this at all. Software organizations can achieve massive gains in velocity by adopting DevOps practices.

Teams and organizations can follow a similar journey to those chronicled in DevOps lore like The Phoenix Project (shameless plug: see my book summary) and The Unicorn Project, and described in much greater detail from tactical (The DevOps Handbook) and strategic (Accelerate) perspectives. There aren’t many reasons not to investigate at this point, so check those sources out. If you’re curious about a handy set of principles for approaching DevOps, learn about The Three Ways (another shameless plug: head on over to my article Applying the Three Ways of DevOps to Accelerate Your Organization).

Additionally, this DZone article offers even broader approaches.

Solution: Dedicate Regular Time to Technical Debt Cleanup

Technical debt is best dealt with by the people closest to the code: the developers. Often this work is treated the same as some feature that you can get away with prioritizing later, secretly hoping it slips out of the bottom of your backlog into oblivion. Tech debt is different —it’s something that your team has to build habits around. I suggest starting with 10% to 20% of developer time. That translates to 1/2 day to 1 day per week.

There are exceptions, of course. For example, you may need to plan bigger chunks of work, like implementing a new library, refactoring a large part of your system, or extracting part of your codebase into microservices. However, to be in a position to entertain thoughts of that sort of technical debt, you’ll need to ensure that all of the little things taken care of:

Dependencies should be up to date and relatively easy to upgrade. Your tests should be up to date, and you should have decent coverage on your unit tests. If you don’t have some level of acceptance testing in place, you’re probably opening the door to regressions, which will suck up more of your time. So automate aggressively, or plan to test manually.

A Side-Note on Refactoring

Some teams confuse rewriting with refactoring. Refactoring is when you restructure your code (either production or test code) to address some non-functional requirement (e.g., readability) without changing the behaviour. That seems relatively straightforward. The subtlety that needs some attention is that substantially changing your implementation without the backing of solid automated tests isn’t really in the spirit of refactoring. You might as well call it what it is: a rewrite. If you engage in a rewrite, it’s an excellent idea to make sure you do so and put in the missing automated tests.

Conclusion

As hindsight bias teaches us, it’s not so easy to know that things are heading downhill when you’re in the thick of development. Don’t allow yourself or your team to become victims of false confidence. The boost you get from the feeling of early progress is excellent, but it can come at a price if you want to avoid the crushing lows that result if you ignore technical debt. Healthy skepticism mixed with curiosity is a vital outlook to develop.

With the accelerated pace of DevOps workflows that are quickly becoming the norm, these skills are even more critical. Not just individual developers and teams, but entire software organizations. We must avoid burnout at all costs to prevent a downward spiral from starting.

The bright side is that there is an abundance of knowledge around the practices, philosophies, tools, and research easily accessible. At this point, it’s a matter of professional responsibility to at least become familiar with it. Check out the material in the links above and engage people in your organization. Try different things and pay attention to what thought leaders and look for advice in the stories coming out of organizations finding success in their digital transformations. Best of luck in your journey!

Disclaimer

I’m an employee of IBM. The views expressed in this blog are mine alone. They don’t necessarily reflect the positions, strategies, or opinions of the company.