What is Code Coverage?

What is Code Coverage?

A measure that describes the degree to which code has been tested. The higher the Code Coverage of a project, the more thoroughly tested it has been, and thus the chance of it containing bugs is (usually) smaller.

How it works

There are several ways that Code Coverage can be implemented (and usually you won’t have to know how it really works unless you’re creating a coverage tool).

One of the possibilities is to implement checks throughout the code, run the test suite, and then analysing how many of those checks were actually run.

How is Code Coverage Calculated?

There are several ways of calculating coverage. Usually, coverage tools will employ a few of them and report with the individual and average scores for your code.

The four main metrics used are:

statement coverage — how many of the statements in the program have been executed?

branch coverage — how many of the branches of each control structures have been executed?

condition coverage — for each boolean sub-expression, were they tested for both true and false?

function/subroutine coverage — how many of the declared functions have been called during testing?

To better understand how these metrics work let’s take a look at an example (in Python):

def parity(number):
  if (number % 2 == 0):
    return 'even'
  else:
    return 'odd'

The function above takes a number and determines whether it is odd or even.

A test such as:

assertEqual( parity(2), ‘even’ )

Would result in a function coverage of 100% (because all of the functions we have — just the one — are being called with the test), but statement coverage of only 50%. To get the statement coverage up to 100% you’d need to test with an odd number as well so that the else branch would also be included in the test.

Other metrics

While the above are indeed the four main metrics, there’s also decision coverage (a combination of function coverage and branch coverage), multiple condition coverage (where all combinations of conditions inside each decision have to be tested), linear code sequence and jump (LCSAJ) coverage, path coverage, entry/exit coverage, loop coverage and stage coverage, among others.

One particularly interesting metric is Parameter Value Coverage, where all common values (or, rather, corner cases) for each parameter should be tested.

For instance, if you have a function that receives a list, that same function should be tested with an empty list, a list with repeated elements, a very long list, etc. The idea is that, if one misses testing one of those cases, bugs could be overlooked.

Testing a function with an expected list could result in a coverage of 100%, but if the function has a bug that only manifests on empty lists, the tests aren’t covering all the possible scenarios.

Why Code Coverage matters

Code Coverage data gives you an important insight on how much of your code is being used in the tests you wrote.

If you have 25% coverage in a function with four branches, for instance, that coverage will let you know that your tests are still missing three of those branches.

The relevance of coverage is not that of having a 100% value to show to other people, but rather to have a value (whatever it may be) and ensure that it doesn’t decrease over time, as that could mean that new code is not being tested.

There’s actually a lot more that can be said about this:

Caveat

However, one should not blindly follow the goal of 100% coverage just for the sake of that number.

While Code Coverage can be considered an indirect measure of quality, it is not necessarily a good metric of how well you are testing your code.

Just because your tests are going through all your lines of code doesn’t mean they’re testing every possible scenario, and thus you code can still be faulty.

As an example, consider this function:

def divide(total, number_of_people):
  return total/number_of_people

Let’s say we use that function every day to see how much each of the people you usually have lunch with has to pay. Let’s also say you get those parameters from a web service.

It’s very easy to reach 100% code coverage, as you may imagine.

And then, finally, Saturday comes.

And you have lunch at home.

And now you have a total of 0 to split between 0 people.

Code Coverage was 100%, but you still hadn’t tested properly.

That being said, Code Coverage is still important, because it gives you an idea of how much you’re testing (or not testing), but it’s very important to know what and how to test as well.

Code Coverage tools

There are many Code Coverage tools out there.

Here are some of them worth mentioning (all of them supported by Codacy):

• Java: Cobertura and JaCoCo
• JavaScript: istanbul, js.coverage and blanket.js
• PHP: Clover and PHPUnit
• Python: coverage.py
• Ruby: SimpleCov
• Scala: Scoverage and Cobertura

Integration of Code Coverage Tools with Codacy

Codacy allows you to integrate your favourite tool for Code Coverage in your project. The instructions are simple and the result is a nice widget in your dashboard that will display your current coverage and a few files with the lowest coverage in your project.

Integrating Code Coverage with Codacy will also give you a graphic of how your coverage has been evolving through time.

Edit: We just published an ebook: “The Ultimate Guide to Code Review” based on a survey of 680+ developers. Enjoy!

About Codacy

Codacy is used by thousands of developers to analyze billions of lines of code every day!

Getting started is easy – and free! Just use your  GitHub, Bitbucket or Google account to sign up.

GET STARTED