How 680 Developers Say We Can Increase Code Quality

TL;DR: We recently surveyed over 680 developers and published a first set of insights here. Today we discuss one specific aspect: the impact or code reviews on developer productivity. We find that code reviews increase code quality and developer productivity on several dimensions.

Disclosure: Codacy, the automated code reviews platform, has previously sponsored Hacker Noon. For Hacker Noon readers, they’re offering 15% off using this code: HACKERNOON.

Step 1. Identifying the most important practice to improve code quality

It is widely assumed that code reviews have a number of benefits. A casual browse of the literature covering the topic of code reviews reveals that they have benefits such as:

• improving code quality;
• facilitating team collaboration;
• homogenizing code standards and code styles;
• identifying bugs early in the development process;
• onboarding new developers and spreading best practices and knowledge within the organization.

However we wanted to contrast the intuition with hard and cold numbers.

The first step was to ask an open question: what change to their development process had the biggest impact to code quality? Developers were free to provide as many answers as they wanted. The answers were then labelled and categorized, as shown in the chart below.

Question 1: what change to your development process had the biggest impact on code quality?

Result: 1 in 5 respondents answered that code reviews was the single most important activity to to improve code quality.

Of all the answer provided by the developers, code reviews was by far the most prevalent answer, with nearly 1 in 5 (specifically 19%) suggesting that code reviews was the single most important activity to to improve code quality.

Step 2. Measuring the impact of code reviews on developer productivity

There’s an obvious trade-off between time spent reviewing code and time spent coding new features. So we asked developers what proportion of their time they spend fixing bugs.

Question 2: what proportion of your time do you spend fixing bugs?

Results:

The above table is pretty self-explanatory. If you assume that a week consists of 40 hours, developers who work in companies where the code is reviewed systematically before deployment spend 4 hours less per week fixing bugs.

But it gets better.

Because the other side of the coin is whether the time saved fixing bugs actually translates in the more fun, productive stuff — that is, building new features. So we asked the following question:

Question 3: what proportion of your time do you spend building new features?

Results:

It turns out that people who who review code spend 4.4 hours per week more building new features (again, assuming a 40-hour week).

Step 3. Measuring the impact of code reviews on code quality

Having established that developers who have their code reviewed end up spending 10% more of their time on building new features and 10% less on fixing bugs, the next set of question relates to code quality: how much of an improvement do code reviews contribute?

To find out, we ask two questions, starting with…

Question 4: How do you rate code quality in your project?

Here we asked point blank developers how they rated the quality of their project, on a scale from 1 (very bad) to 5 (very good). The average scores are displayed in the table below.

Those did not do code reviews had an average score below the median (i.e. 3), whereas those who did do code reviews had an average quality score _above_the median 3. This code quality score is obviously highly subjective, so we asked another question:

Question 5: do you need more time to maintain code?

Developers here asked whether they needed more time to maintain code, and had only possible options: yes, or no. The result is as follow:

Whether or not the code is reviewed, in general there was a feeling that more time was needed to review code (in fact, time pressures were one of the main complaints that developers reported). But again, the difference between those who do code reviews and those who don’t is stark: over three-quarters of developers who didn’t have their code reviewed felt that they needed more time to maintain the code, compared to 59% for those who did.

Final thoughts

Our research confirms what is known intuitively by most developers. It also complements insights from a variety from case studies first reported by McConnell’s “Code Complete” book. We’ll conclude this blog post by adding a few quotes from the book:

“… software testing alone has limited effectiveness — the average defect detection rate is only 25 percent for unit testing, 35 percent for function testing, and 45 percent for integration testing. In contrast, the average effectiveness of design and code inspections are 55 and 60 percent. Case studies of review results have been impressive:

* In a software-maintenance organization, 55 percent of one-line maintenance changes were in error before code reviews were introduced. After reviews were introduced, only 2 percent of the changes were in error. When all changes were considered, 95 percent were correct the first time after reviews were introduced. Before reviews were introduced, under 20 percent were correct the first time.

* In a group of 11 programs developed by the same group of people, the first 5 were developed without reviews. The remaining 6 were developed with reviews. After all the programs were released to production, the first 5 had an average of 4.5 errors per 100 lines of code. The 6 that had been inspected had an average of only 0.82 errors per 100. Reviews cut the errors by over 80 percent.

* The Aetna Insurance Company found 82 percent of the errors in a program by using inspections and was able to decrease its development resources by 20 percent.

* IBM’s 500,000 line Orbit project used 11 levels of inspections. It was delivered early and had only about 1 percent of the errors that would normally be expected.

* A study of an organization at AT&T with more than 200 people reported a 14 percent increase in productivity and a 90 percent decrease in defects after the organization introduced reviews.

* Jet Propulsion Laboratories estimates that it saves about $25,000 per inspection by finding and fixing defects at an early stage.”

Further reading

• Seven Truths About Peer Reviews (Karl E. Wiegers). Link.
• Code complete. (Steve McConnell). Link.
• Code Reviews: Just Do It. Link. Coding Horror. Link.

Thanks for reading! If you enjoyed it, hit that heart button below :)

You can also read more about this and other great learnings in our ebook about code reviews:

Click to get the ebook

This story was originally published on the Codacy Blog: Insights for Software Builders.