According to our survey of 680 developers, the code review process had an overall positive impact on developer productivity which we discuss below. We find that code reviews increase code quality and software developer productivity on several dimensions.
Step 1. Identifying the most important practice to improve code quality
It is widely assumed that code reviews have a number of benefits. A casual browse of the literature shows that the code review process has benefits such as:
- improving code quality;
- facilitating team collaboration;
- homogenizing code standards and code styles;
- identifying bugs early in the development process;
- onboarding new developers and spreading best practices and knowledge within the organization.
However we wanted to contrast the intuition with hard and cold numbers.
The first step was to ask an open question: what change to their development process had the biggest impact to code quality? Developers were free to provide as many answers as they wanted. The answers were then labelled and categorized, as shown in the chart below.
Question 1: what change to your development process had the biggest impact on code quality?
Result: 1 in 5 respondents answered that code reviews was the single most important activity to to improve code quality.
Of all the answer provided by the developers, code reviews was by far the most prevalent answer, with nearly 1 in 5 (specifically 19%) suggesting that code reviews was the single most important activity to to improve code quality.
Step 2. Measuring impact of code review process on developer productivity
There’s an obvious trade-off between time spent reviewing code and time spent coding new features. So we asked developers what proportion of their time they spend fixing bugs.
Question 2: what proportion of your time do you spend fixing bugs?
Results:
The above table is pretty self-explanatory. If you assume that a week consists of 40 hours, developers who work in companies where the code is reviewed systematically before deployment spend 4 hours less per week fixing bugs.
But it gets better.
Because the other side of the coin is whether the time saved fixing bugs actually translates in the more fun, productive stuff — that is, building new features. So we asked the following question:
Question 3: what proportion of your time do you spend building new features?
Results:
It turns out that people who who review code spend 4.4 hours per week more building new features (again, assuming a 40-hour week).
Step 3. Measuring the impact of code reviews on code quality
Having established that developers who have their code reviewed end up spending 10% more of their time on building new features and 10% less on fixing bugs, the next set of question relates to code quality: how much of an improvement do code reviews contribute?
To find out, we ask two questions, starting with…
Question 4: How do you rate code quality in your project?
Here we asked point blank developers how they rated the quality of their project, on a scale from 1 (very bad) to 5 (very good). The average scores are displayed in the table below.
Those did not do code reviews had an average score below the median (i.e. 3), whereas those who did do code reviews had an average quality score above the median 3. This code quality score is obviously highly subjective, so we asked another question:
Question 5: do you need more time to maintain code?
Developers here asked whether they needed more time to maintain code, and had only possible options: yes, or no. The result is as follow:
Whether or not the code is reviewed, in general there was a feeling that more time was needed to review code (in fact, time pressures were one of the main complaints that developers reported). But again, the difference between those who do code reviews and those who don’t is stark: over three-quarters of developers who didn’t have their code reviewed felt that they needed more time to maintain the code, compared to 59% for those who did.
Final thoughts
Our research confirms what is known intuitively by most developers. It also complements insights from a variety from case studies first reported by McConnell’s “Code Complete” book. We’ll conclude this blog post by adding a few quotes from the book:
“… software testing alone has limited effectiveness — the average defect detection rate is only 25 percent for unit testing, 35 percent for function testing, and 45 percent for integration testing. In contrast, the average effectiveness of design and code inspections are 55 and 60 percent. Case studies of review results have been impressive:
* In a software-maintenance organization, 55 percent of one-line maintenance changes were in error before code reviews were introduced. After reviews were introduced, only 2 percent of the changes were in error. When all changes were considered, 95 percent were correct the first time after reviews were introduced. Before reviews were introduced, under 20 percent were correct the first time.
* In a group of 11 programs developed by the same group of people, the first 5 were developed without reviews. The remaining 6 were developed with reviews. After all the programs were released to production, the first 5 had an average of 4.5 errors per 100 lines of code. The 6 that had been inspected had an average of only 0.82 errors per 100. Reviews cut the errors by over 80 percent.
* The Aetna Insurance Company found 82 percent of the errors in a program by using inspections and was able to decrease its development resources by 20 percent.
* IBM’s 500,000 line Orbit project used 11 levels of inspections. It was delivered early and had only about 1 percent of the errors that would normally be expected.
* A study of an organization at AT&T with more than 200 people reported a 14 percent increase in productivity and a 90 percent decrease in defects after the organization introduced reviews.
* Jet Propulsion Laboratories estimates that it saves about $25,000 per inspection by finding and fixing defects at an early stage.”
