Code Coverage: A Complete Guide

Building software is similar to building a skyscraper. It’s a meticulous process that demands precision, foresight, and attention to detail. Imagine constructing a towering building without periodically inspecting its internal framework or ensuring the stability of each intricate component. It's a recipe for disaster, right?

Much like a construction site requires diligent quality checks to guarantee the safety and reliability of a building, software development demands a robust testing strategy. Code coverage emerges as the vigilant inspector in this realm, scrutinizing every nook and cranny of your codebase to unveil potential vulnerabilities and weaknesses.

Every deployment of bad code can cost your company thousands, if not millions, in losses. Bad code can cost you customers and leave vulnerabilities in your software for cybercriminals to attack, potentially compromising sensitive secrets in your codebase.

According to CISQ’s 2022 report, poor software quality cost US software companies at least $2.41 trillion last year, resulting in an additional increase in accumulated technical debt of about $1.5 trillion as well.

Why isn’t code testing a widely accepted requirement if it is so important? Writing and performing tests is a time and resource-intensive process. It takes a lot of work and effort to build testing infrastructures, especially for legacy systems. Even if you’re working with brand new code, early-stage software companies often skip testing to make the software development process faster and get their product to market as quickly as possible. 

Teams prioritizing system security and stability must also prioritize properly testing their code. 

What is Code Coverage?

Code coverage is a software development metric used to measure the extent to which software code is executed during testing. Coverage is a quantitative measure that indicates the percentage of code lines, branches, statements, or other structural elements covered by a test suite. In simpler terms, it helps developers understand how much of their codebase is exercised by their tests.

Code coverage tools usually express the metric as a percentage, showing you the percentage of successfully validated lines of code in your test procedures, helping you to understand how thoroughly you’re testing your code. 

Why Is Code Coverage Important?

The importance of code coverage lies in its ability to provide insights into the quality and effectiveness of a test suite. Code coverage metrics can help your team in: 

• Identifying uncovered code that has not been tested and could contain bugs or potential issues not addressed by the test suite.
  
  
• Improving code maintenance or refactoring activities. When code changes are made, code coverage reports can help ensure that any modifications introduced do not inadvertently introduce new issues.
  
  
• Increasing testing effectiveness by uncovering whether tests are reaching all parts of the code or if certain code paths are being neglected.

• Identifying unnecessary tests and eliminating them to increase the speed of testing and make your code lighter. 

It's important to note that while code coverage is a valuable metric, achieving 100% coverage does not guarantee a bug-free application. It is just one of many tools and practices in a developer's toolkit for ensuring software and code quality. The emphasis should be on meaningful tests that cover a variety of scenarios, including edge cases and potential error conditions.

How is Code Coverage Calculated?

Imagine you have a test suite for your code. Before your code runs in these tests, the system adds counters to keep tabs on which lines of code are actually used. This information is stored in a central place that keeps track of all the coverage details. This whole setup is called "instrumentation." It's like putting trackers on specific parts of your code to see how much of it gets used when your tests run.

Coverage tools typically use the following criteria to determine how your code was exercised while executing your test suite and whether it was executed at all:

Function Coverage 

This code coverage metric measures the percentage of functions or subroutines executed during testing. It shows how many code functions have been called out at least once during testing. 

Achieving 100% function coverage ensures that every defined function has been invoked at least once in the test suite.

Example:

# File: calculator.py
def add(a, b):
   return a + b

def subtract(a, b):
   return a - b

In this example, achieving 100% function coverage would mean executing both ‘add’ and ‘subtract’ functions in your test suite.

Statement Coverage 

Statement coverage focuses on the execution of individual statements within functions. It ensures that every line of code has been traversed during testing. Full statement coverage is important for identifying dead code (code that is never executed) and ensuring that each part of the code is reachable and tested. It also helps identify missing statements, and unused statements and branches. 

Example:

# File: calculator.py

def multiply(a, b):

   result = a * b

   print(result)

   return result

Statement coverage involves executing both the calculation (‘result = a * b’) and the ‘print(result)’ statement.

Branch Coverage

In coding, a branch is the point in the code that enables you to direct the program flow to one or more paths. This type of coverage extends the idea of statement coverage by focusing on decision points in the code. Branch coverage measures the percentage of branches that have been taken during testing. Full branch coverage is essential for ensuring that all possible decision outcomes are considered and tested.

Example:

# File: number_classifier.py

def is_positive(num):
   if num > 0:
       return True
   else:
       return False

To achieve 100% branch coverage, you need test cases that cover both the ‘if’ and ‘else’ branches.

Condition Coverage

Conditional coverage delves into the evaluation of Boolean conditions within functions. Boolean conditions refer to expressions or statements in programming that evaluate to either ‘True’ or ‘False’. In programming, Boolean values are often associated with logical operations and comparisons. These conditions are fundamental for decision-making in control structures like ‘if’ statements and loops. Condition coverage ensures that each condition has been tested both as true and false. This is valuable for catching potential issues related to conditional logic and ensuring the correctness of decision-making processes.

Example:

# File: voting_age_checker.py

def cab_vote(age):
   return age >= 18

Achieving condition coverage involves testing with inputs where ‘age’ is both greater than and less than 18.

Multiple condition decision coverage (MC/DC) is a more stringent form of condition coverage that ensures each condition independently affects the decision outcome. This level of coverage is particularly important in safety-critical systems, where the correct functioning of complex decision logic is crucial.

Example:

# File: loan_approval.py

def approve_loan(income, credit_score):
   return income > 50000 and credit_score > 700

MC/DC coverage requires test cases where changing either the ‘income’ or ‘credit_score’ independently influences the decision.

Outside of these four most common ones, other types of code coverage teams often test for include line coverage and parameter value coverage. Line coverage measures the lines of code that have been executed during testing. It provides a broad overview of the code's test coverage and helps identify gaps in testing. However, it may not catch certain nuances, such as partial execution of lines.

Parameter value coverage ensures that functions are tested with various input values. This is crucial for uncovering issues related to parameter handling, boundary conditions, and overall robustness of the function across different input scenarios.

Incorporating a combination of these coverage types in a test suite provides a multifaceted approach to ensuring improved code quality.

Now, let’s create an example of a more complex function, one that determines the eligibility of a person for a discount based on various conditions. This function has nested statements and multiple conditions, making it more complex. 

# File: discount_calculator.py

def calculate_discount(age, income, has_membership):
   """
   Calculates the discount eligibility based on age, income, and membership.
   """
   discount = 0

   if age >= 18:
       if income > 50000:
           discount = 10
           if has_membership:
               discount += 5
       elif income > 30000:
           discount = 5
   else:
       discount = 0

   return discount

Now, let's create a set of tests for this function.

# File: test_discount_calculator.py

from discount_calculator import calculate_discount

def test_eligible_for_discount():
   result = calculate_discount(25, 60000, True)
   assert result == 15

def test_eligible_for_partial_discount():
   result = calculate_discount(30, 40000, True)
   assert result == 5

def test_not_eligible_for_discount():
   result = calculate_discount(16, 35000, False)
   assert result == 0

In this example, we have three test cases covering different scenarios. However, achieving 100% coverage for all possible code paths may be difficult, especially if the function's logic is complex. This is because testing all possible combinations of conditions and branches becomes cumbersome. 

Recognizing that you’re struggling to reach 100% coverage for a function because tests are too complicated likely means that the function is probably too complicated and should probably be reassessed.

Benefits and Limitations of Measuring Code Coverage

While achieving 100% coverage in every category doesn't guarantee the absence of bugs, it significantly reduces the likelihood of undiscovered issues. Code coverage percentages offer a measurable value of code quality that can be presented to stakeholders who might not be involved in day-to-day development processes. 

These metrics help teams identify code that isn’t being tested and identify parts of the code that might require additional testing. Code coverage tests can also identify dead code that no longer serves any purpose and can be safely removed from the codebase. 

Achieving high coverage percentages can also speed up production by showing developers what portions of code require more attention.  According to our 2024 State of Software Quality report, 58% of developers say not having enough time is the single most common challenge faced during code reviews. Code coverage also promotes better code understanding, maintenance, and collaboration among developers. 

However, it’s important to remember that while coverage coverage helps you evaluate code quality, widespread coverage does not guarantee code quality. 

A test suite could have excellent coverage statics, but quality won’t improve if the tests are inefficient or poorly written. With code coverage, you’re measuring the execution of code, not the actual quality of the tests and the overall design and implementation of your test system. 

Code Coverage vs. Test Coverage

"Code coverage" and "test coverage" are terms often used interchangeably but have different meanings and outcomes.  Code coverage is about the amount of code executed. It tells your team what areas of code have and haven’t been executed, while test coverage informs your team of the risks that have been evaluated and examined. 

Code coverage encompasses various metrics measuring the extent to which a test suite executes your source code. It provides information about which lines, branches, conditions, or functions in your code have been executed during testing. 

It’s important to remember that code coverage is not limited to just tests; it includes any method of code execution, such as manual testing or other forms of testing like static code analysis.

Test coverage refers explicitly to the proportion of your codebase exercised by a set of tests. It quantifies how well your tests exercise different parts of your code. Therefore, test coverage is a component of code coverage, providing a more focused perspective on the effectiveness of your test suite in testing the codebase. 

Test coverage can be achieved through various testing techniques, each serving a specific purpose in ensuring the quality and reliability of software:

• Unit tests involve testing individual units or components of a software application in isolation. Units are the smallest testable parts, such as functions or methods. 

• Responsive tests focus on validating that a web application or website displays correctly and functions well across different devices and screen sizes.

• Cross-browser tests verify that web applications or websites work across browsers to ensure compatibility and consistent behavior.

• Integration tests verify interactions between various components or modules of a system.

• Acceptance tests evaluate whether a software application meets specified requirements and is ready to deploy. 

• Regression tests ensure that new code changes do not negatively impact existing functionality.

Development teams measure test coverage to ensure critical business requirements are prioritized via testing for a variety of coverage types: 

• Requirement coverage ensures that the testing process aligns with the specified functional and non-functional requirements, validating that the software meets its intended purpose.

• Risk coverage directs testing efforts toward areas of the codebase that pose potential risks, helping teams prioritize testing activities and mitigate critical issues.

• Product coverage thoroughly evaluates the entire software product, ensuring that different features and components are adequately tested.

By considering these aspects in addition to traditional metrics like line, branch, and statement coverage, development teams can adopt a more thorough and nuanced approach to test coverage, addressing different dimensions of software quality and reliability.

Code Coverage Tools

No matter what your team’s preferred programming language is, plenty of code coverage tools are available to help you gather data on the tests you’re running, analyze them, and generate coverage reports. 

Once you run and execute your tests, these tools calculate the percentage of code executed, after which they generate a code coverage analysis report.

For quick reference, here is a list of popular code coverage tools by programming languages that are worth checking out:

• Java: JaCoCo, Cobertura
• Javascript: Istanbul, jscoverage
  
• Ruby: SimpleCov, undercover
  
• Python: pytest-cov, Coverage.py
  
• C#: Coverlet, NCover
  
• C++: Parasoft Jtest, Testwell CTC++
  
• Rust: Tarpaulin, grcov 
  
• PHP: Xdebug, PHPUnit
  
• Dart: test-coverage, coverage
  
• Scala: scoverage, scct

Centralize Your Code Coverage Insights 

Once you’ve picked the coverage tools that work best for your team, the next step is finding a way to monitor your code coverage reports easily. This can be especially daunting for large software companies with many repositories to monitor. 

Codacy Coverage makes this easy, providing a repository dashboard with a simple and effective way to monitor all your coverage. All you need to do is generate coverage reports for commits on your CI/CD workflow and then upload the coverage data to Codacy.

The dashboard provides valuable data at a glance, including a code coverage percentage, coverage evolution reports, and a list of open pull requests for every repository.

Start a free trial today to get a closer look at how Codacy’s coverage dashboard works and how easy analyzing and reporting on code coverage can truly be.