Home Developer Three top points on code coverage

Three top points on code coverage

Author

Date

Category

Below are descriptions of three top points regarding code coverage including important industry data points so we get a sense of what real companies are doing when it comes to code coverage.
You can also join the discussion on
reddit and check our ebook: the ultimate guide to code reviews.
We also just 
released a security dashboard for your team in Codacy.

Best practices dictate that you should track code coverage. Code coverage helps you assess the proportion of code that was tested and (more importantly) not tested.

Why is it important to track coverage? Our users report that coverage is important because it is the confidence level of code being correct and with less bugs. We’ve written previously about code coverage, this is a follow up based on industry data.

In this article we share some evidence of how different companies do code coverage, based on a survey of 680 companies about their code quality practices. We’ll answer three questions:

  1. Who tracks code coverage? What happens when you don’t?
  2. What thresholds are used for code coverage?
  3. What workflow rules are implemented with code coverage?

Without further ado…

Who tracks code coverage? What happens when you don’t?

Let’s first look at some of the numbers from our research.

Code coverage breakdown
Code coverage breakdown

Half of the companies track code coverage. We’ll see in a second how that impacts code quality and other parts of development.

This means that half of the companies cannot trust their code to be without bugs.

The bigger the company the more likely it is to track code coverage, as shown in the chart below.

Coverage tracking by company size
Coverage tracking by company size

You can see that the larger the company the more likely it is to be tracking code coverage.

Not surprisingly, the older the company the more likely it is to track code coverage too.

Coverage tracking by company age
Coverage tracking by company age

So the first major question we need to ask is: what is the impact of not tracking coverage?

Here is a quick summary of the impact:

  • 68% of people who do not monitor code coverage feel they should have more time to maintain code.
  • 59% of those who do monitor code coverage feel they should have more time to maintain code.
  • People who monitor code coverage perceive their code quality to be 7,2% better

Conclusion: although subtle, there are benefits to tracking code coverage.
People who track code coverage feel that the quality of their code is higher, and don’t feel they need to spend more time reviewing code. This the likely reason why more mature companies, which tend to be larger and older, implement code coverage systematically.

Code coverage thresholds?

Code coverage at any given time is really just a number. It’s the evolution of that number that matters. Code coverage that goes down is an indication that new code is not being tested therefore the confidence level is going down. If it’s going up, normally it’s a good sign.

Minimum code coverage threshold to accept a Pull Request? Most companies don’t

Only 34% of companies require a minimum code coverage threshold to accept a pull request.

What is surprising is the difference between those 34% and 65%.

Here is the real difference between those two groups:

  • Companies that require a minimum code coverage threshold perceive their code quality to be 6,4% better.
  • Companies that don’t enforce code coverage thresholds say their biggest problem is technical debt.
  • On the other hand, quality comes at a cost: companies that enforce code coverage thresholds say their biggest problem is speed of delivery.

Another important question is: what is the ideal threshold?

Code coverage popularity
Code coverage popularity

The vast majority of companies that have a code coverage target have set it at 80%. This means the companies that do track code coverage actually set a very high bar with each pull request.

Conclusion: having code coverage thresholds increased perceived code quality and decreased time spent on tackling technical debt. Companies that required this threshold set the bar, on average, at 80%.

Code coverage workflow rules?

Our final consideration is about code coverage workflow rules.
Do companies require code code coverage to increase whenever new code is introduced? Or do they enforce other rules, for example requiring code coverage not to go down?

First question is: how many companies have code coverage workflow rules?

Less than 20% of companies have rules for code coverage in their workflows.

What are the rules that companies define?

We’ve asked about four major workflow:

  • Coverage must be same or increase
  • Coverage must increase
  • Coverage must not fall
  • Coverage must pass threshold

Here is the popularity of these rules:

Final thoughts

Code coverage is only as good as the quality of the tests that drive it. Setting a 100% target may tempt developers to game the system by writing unit tests that just increase coverage, at the cost of actually testing the application meaningfully. This is why the majority of the respondents reported a threshold of 80% approx, the most popular workflow rules being “coverage must not fall” or “coverage must pass threshold”.

What are your thoughts on these code coverage data points? Let me know!

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Subscribe to our newsletter

To be updated with all the latest news, offers and special announcements.

Recent posts

Pair programming at Codacy and why we do it

Pair programming, also known as pairing or “dynamic duo” model is not a new concept, and it was pioneered by C/C++ guru...

Enhanced security for C++, Java, and Scala with Clang-Tidy and SpotBugs

As part of our effort to continue expanding our language support, we are excited to announce the support of two new tools...

Improve the efficiency of your remote engineering team

COVID-19 hit the ground running and the world felt the impact. Although tech companies seemed to be ahead of the curve by...

Further Enterprise security analysis for Scala

We’re excited to announce the latest addition to our suite of security analysis: Spotbugs. SpotBugs is a program which...

Free Codacy Pro account to fight COVID-19

Our hearts go out to everyone who has been directly or indirectly impacted by the global coronavirus (COVID-19) pandemic. We are committed...