Code Coverage: A Complete Guide

Building software is similar to building a skyscraper. It’s a meticulous process that demands precision, foresight, and attention to detail. Imagine constructing a towering building without periodically inspecting its internal framework or ensuring the stability of each intricate component. It's a recipe for disaster, right?

Much like a construction site requires diligent quality checks to guarantee the safety and reliability of a building, software development demands a robust testing strategy. Code coverage emerges as the vigilant inspector in this realm, scrutinizing every nook and cranny of your codebase to unveil potential vulnerabilities and weaknesses.

Every deployment of bad code can cost your company thousands, if not millions, in losses. Bad code can cost you customers and leave vulnerabilities in your software for cybercriminals to attack, potentially compromising sensitive secrets in your codebase.

According to CISQ’s 2022 report, poor software quality cost US software companies at least $2.41 trillion last year, resulting in an additional increase in accumulated technical debt of about $1.5 trillion as well.

Why isn’t code testing a widely accepted requirement if it is so important? Writing and performing tests is a time and resource-intensive process. It takes a lot of work and effort to build testing infrastructures, especially for legacy systems. Even if you’re working with brand new code, early-stage software companies often skip testing to make the software development process faster and get their product to market as quickly as possible. 

Teams prioritizing system security and stability must also prioritize properly testing their code. 

What is Code Coverage?

Code coverage is a software development metric used to measure the extent to which software code is executed during testing. Coverage is a quantitative measure that indicates the percentage of code lines, branches, statements, or other structural elements covered by a test suite. In simpler terms, it helps developers understand how much of their codebase is exercised by their tests.

Code coverage tools usually express the metric as a percentage, showing you the percentage of successfully validated lines of code in your test procedures, helping you to understand how thoroughly you’re testing your code. 

Why Is Code Coverage Important?

The importance of code coverage lies in its ability to provide insights into the quality and effectiveness of a test suite. Code coverage metrics can help your team in: 

• Identifying uncovered code that has not been tested and could contain bugs or potential issues not addressed by the test suite.
  
  
• Improving code maintenance or refactoring activities. When code changes are made, code coverage reports can help ensure that any modifications introduced do not inadvertently introduce new issues.
  
  
• Increasing testing effectiveness by uncovering whether tests are reaching all parts of the code or if certain code paths are being neglected.
  
  
• Identifying unnecessary tests and eliminating them to increase the speed of testing and make your code lighter. 

It's important to note that while code coverage is a valuable metric, achieving 100% coverage does not guarantee a bug-free application. It is just one of many tools and practices in a developer's toolkit for ensuring software and code quality. The emphasis should be on meaningful tests that cover a variety of scenarios, including edge cases and potential error conditions.

How is Code Coverage Calculated?

Imagine you have a test suite for your code. Before your code runs in these tests, the system adds counters to keep tabs on which lines of code are actually used. This information is stored in a central place that keeps track of all the coverage details. This whole setup is called "instrumentation." It's like putting trackers on specific parts of your code to see how much of it gets used when your tests run.

Coverage tools typically use the following criteria to determine how your code was exercised while executing your test suite and whether it was executed at all:

Function Coverage 

This code coverage metric measures the percentage of functions or subroutines executed during testing. It shows how many code functions have been called out at least once during testing. 

Achieving 100% function coverage ensures that every defined function has been invoked at least once in the test suite.

Example:

# File: calculator.py
def add(a, b):
   return a + b

def subtract(a, b):
   return a - b

In this example, achieving 100% function coverage would mean executing both ‘add’ and ‘subtract’ functions in your test suite.

Statement Coverage 

Statement coverage focuses on the execution of individual statements within functions. It ensures that every line of code has been traversed during testing. Full statement coverage is important for identifying dead code (code that is never executed) and ensuring that each part of the code is reachable and tested. It also helps identify missing statements, and unused statements and branches. 

Example:

# File: calculator.py

def multiply(a, b):

   result = a * b

   print(result)

   return result

Statement coverage involves executing both the calculation (‘result = a * b’) and the ‘print(result)’ statement.

Branch Coverage

In coding, a branch is the point in the code that enables you to direct the program flow to one or more paths. This type of coverage extends the idea of statement coverage by focusing on decision points in the code. Branch coverage measures the percentage of branches that have been taken during testing. Full branch coverage is essential for ensuring that all possible decision outcomes are considered and tested.

Example:

# File: number_classifier.py

def is_positive(num):
   if num > 0:
       return True
   else:
       return False

To achieve 100% branch coverage, you need test cases that cover both the ‘if’ and ‘else’ branches.

Condition Coverage

Conditional coverage delves into the evaluation of Boolean conditions within functions. Boolean conditions refer to expressions or statements in programming that evaluate to either ‘True’ or ‘False’. In programming, Boolean values are often associated with logical operations and comparisons. These conditions are fundamental for decision-making in control structures like ‘if’ statements and loops. Condition coverage ensures that each condition has been tested both as true and false. This is valuable for catching potential issues related to conditional logic and ensuring the correctness of decision-making processes.

Example:

# File: voting_age_checker.py

def cab_vote(age):
   return age >= 18

Achieving condition coverage involves testing with inputs where ‘age’ is both greater than and less than 18.

Multiple condition decision coverage (MC/DC) is a more stringent form of condition coverage that ensures each condition independently affects the decision outcome. This level of coverage is particularly important in safety-critical systems, where the correct functioning of complex decision logic is crucial.

Example:

# File: loan_approval.py

def approve_loan(income, credit_score):
   return income > 50000 and credit_score > 700

MC/DC coverage requires test cases where changing either the ‘income’ or ‘credit_score’ independently influences the decision.

Outside of these four most common ones, other types of code coverage teams often test for include line coverage and parameter value coverage. Line coverage measures the lines of code that have been executed during testing. It provides a broad overview of the code's test coverage and helps identify gaps in testing. However, it may not catch certain nuances, such as partial execution of lines.

Parameter value coverage ensures that functions are tested with various input values. This is crucial for uncovering issues related to parameter handling, boundary conditions, and overall robustness of the function across different input scenarios.

Incorporating a combination of these coverage types in a test suite provides a multifaceted approach to ensuring improved code quality.

Now, let’s create an example of a more complex function, one that determines the eligibility of a person for a discount based on various conditions. This function has nested statements and multiple conditions, making it more complex. 

# File: discount_calculator.py

def calculate_discount(age, income, has_membership):
   """
   Calculates the discount eligibility based on age, income, and membership.
   """
   discount = 0

   if age >= 18:
       if income > 50000:
           discount = 10
           if has_membership:
               discount += 5
       elif income > 30000:
           discount = 5
   else:
       discount = 0

   return discount

Now, let's create a set of tests for this function.

# File: test_discount_calculator.py

from discount_calculator import calculate_discount

def test_eligible_for_discount():
   result = calculate_discount(25, 60000, True)
   assert result == 15

def test_eligible_for_partial_discount():
   result = calculate_discount(30, 40000, True)
   assert result == 5

def test_not_eligible_for_discount():
   result = calculate_discount(16, 35000, False)
   assert result == 0

In this example, we have three test cases covering different scenarios. However, achieving 100% coverage for all possible code paths may be difficult, especially if the function's logic is complex. This is because testing all possible combinations of conditions and branches becomes cumbersome. 

Recognizing that you’re struggling to reach 100% coverage for a function because tests are too complicated likely means that the function is probably too complicated and should probably be reassessed.

Benefits of Measuring Code Coverage

Code coverage is a vital metric for assessing the effectiveness of your testing efforts, making it essential to any complete DevSecOps strategy. Here are some benefits of measuring code coverage in your organization.

Improved Code Quality 

While achieving 100% coverage in every category doesn't guarantee the absence of bugs, it significantly reduces the likelihood of undiscovered issues, resulting in a more reliable product for users.

Code coverage sheds light on areas of the code not exercised by tests, which might be crucial to uncovering poor-quality code, dead code, and other errors. In addition, code coverage percentages offer a measurable value of code quality that can be presented to stakeholders who might not be involved in day-to-day development processes. 

Reliability

When code coverage is activated throughout the codebase, there is greater confidence in the integrity of the overall software product. Most code coverage tools provide detailed reports to aid developers in spotting errors and debugging.

Code coverage is like a safety net that strengthens your code, ensuring the software works well in different situations. It helps build a reliable foundation for your software.

Makes Refactoring Easier

Refactoring can be daunting without a clear roadmap. Code coverage removes some risk from the process. It allows developers to confidently restructure existing code without inadvertently breaking functionality, as it clearly shows which parts of the code have already been tested and covered by unit tests.

For example, function coverage is an effective tool for discovering unused or dead code that no longer serves a purpose. It can then be safely removed without altering the functionality.

Guides Developers on Areas Requiring More Testing

Achieving high coverage percentages can also speed up production by showing developers what portions of code require more attention.

Most coverage tools generate reports that show where tests are insufficient or missing entirely and the types of tests needed, enabling developers to identify the most critical or high-risk areas of the application that require more attention. 

Promotes Test-Driven Development

Code coverage is at the core of Test-Driven Development (TDD). It makes sure testing isn’t just something you do later but an essential part of the development process. 

By writing tests before coding, developers are prompted to consider different situations and edge cases from the beginning. This approach results in better design, higher-quality code, and a faster and more effective development cycle.

Facilitates Continuous Integration

Code coverage is a key part of the CI/CD process. Many CI/CD tools run code coverage as part of the pipeline, ensuring that every new code is secure and integrates smoothly. This constant tracking helps keep the integration process efficient and dependable.

Limitations of Measuring Code Coverage

While code coverage is a valuable tool for maintaining clean and secure code, it does have its limitations. Here are a few to consider.

False Sense of Security

Code coverage tracks how much of your code is tested, often setting benchmarks like 80% coverage. However, just because a large percentage of code is covered doesn’t mean the tests are thorough. 

Developers might write tests that only check basic functionality, leaving critical edge cases or failure scenarios untested. This can create a false sense of security, as the code appears well-tested but is not properly vetted.

How to avoid this:

• Focus on writing tests that cover not only common paths, edge cases, and error scenarios. 
  
  
• Pair your code coverage with thorough test reviews to ensure tests are meaningful. 
  
  
• Consider using other testing techniques, like exploratory or fuzz testing, to catch edge cases that unit tests may not cover.

Quality Over Quantity

Focusing too much on achieving a high code coverage percentage can lead to tests that are written just to increase the number of covered lines rather than to truly validate the software’s functionality. 

The goal should always be to write meaningful, well-designed tests that thoroughly examine the software, rather than simply trying to boost coverage numbers. Code coverage should be seen as a helpful tool in a larger testing strategy, not the end goal. 

Code reviews or pair programming can help ensure that tests are thoughtful and comprehensive, as they offer a fresh perspective on the testing approach.

How to avoid this:

• Set clear goals for testing, such as focusing on critical features and scenarios that matter most to your users. 
  
  
• Use coverage as a guideline, not the sole metric of success. 
  
  
• Encourage a culture of quality testing where the focus is on test reliability, design, and effectiveness rather than just boosting numbers. 
  
  
• Code reviews and pair programming can help ensure that tests are meaningful and thorough.

Blind Spots

Code coverage doesn’t capture everything about code quality. It often overlooks critical aspects like error handling, performance issues, and security risks. Even a codebase with high coverage could have serious problems if tests are written just to increase the number of tests rather than to truly check whether the code works as intended. 

A perfect example is when a developer gets a 100% Lighthouse score by creating an incredibly inaccessible website—high coverage doesn’t always mean high-quality code.

How to avoid this:

• Complement code coverage with other quality metrics, such as static code analysis, performance testing, and security audits. 
  
  
• Ensure your testing strategy covers areas like error handling and security vulnerabilities. 
  
  
• Incorporating additional tests like integration or end-to-end tests can help catch issues that unit tests alone might miss. 
  
  
• Regularly run tools like accessibility checkers to ensure overall quality.

Maintenance Challenges

As software grows and changes, keeping up with high code coverage becomes more difficult. Refactoring code, adding new features, or shifting the architecture may break existing tests, requiring a lot of time and effort to update them. 

Solely focusing on maintaining a high coverage percentage can actually slow down the development process and become a hindrance as the codebase evolves.

How to avoid this:

• Don’t treat code coverage as a static goal. Regularly review and refactor tests as the software evolves to keep them aligned with the new code structure.
  
  
• Integrate code coverage monitoring into your continuous integration pipeline to catch gaps early, but don’t let the coverage percentage be the sole focus. 
  
  
• Balance the time spent maintaining tests with the need to deliver new features, ensuring that tests stay valuable without slowing down progress.

Best Practices for Measuring Code Coverage

Adhering to best practices and conventions is key to achieving optimum code coverage. Here are some best practices to follow: 

Select the Best Tool for Your Project

The first step to measuring code coverage is selecting the right tool for the job. Depending on the programming language(s) you're using, there are various tools available to generate code coverage reports. Below are some commonly used options:

• Java: Atlassian Clover, Cobertura, JaCoCo
• JavaScript: Istanbul
• PHP: PHPUnit
• Python: Coverage.py
• Ruby: SimpleCov

Before picking a tool, check its output format and features to see if they align with your project’s needs. Some tools might display results directly in the terminals, while others can generate HTML reports containing the full test coverage report.

Aim For Reasonable Coverage

The level of code coverage you should aim for depends on the complexity and needs of your project. There’s no one-size-fits-all answer. 

Generally, 80% coverage is considered a good benchmark for most applications, balancing coverage with development efficiency. For simple applications, 60-70% should be sufficient, while more complex, mission-critical systems might require 90% or higher coverage for more reliability.

While high code coverage can indicate that most of your code is tested, it doesn’t guarantee your tests are thorough. Focus on writing meaningful tests, especially for critical parts of your application. 

Don’t Skip Unit Testing

Unit tests are designed to verify that individual methods of the classes and components in your application are functioning correctly. They are typically inexpensive to implement, quick to run, and provide confidence that the core of your platform is solid. 

Incorporating unit testing is one good way to improve code coverage; it naturally ensures that your test suite reaches every line of code.

Integrate Coverage Reports into Your CI Pipeline When Ready

Automation is key to efficiently measuring and maintaining code coverage. With automated tools, teams can track and report coverage with every code change. Tools such as JaCoCo (Java), pytest-cov (Python), and Istanbul (JavaScript) offer seamless integration and detailed reporting, allowing teams to monitor coverage regularly without manual intervention.

Once you've set up your Continuous Integration (CI) process, you can begin making test failures contingent on meeting a coverage threshold. However, avoid setting the bar too high—requiring 90% coverage might cause frequent build failures. A more reasonable approach might be to set a lower threshold, like 70%, to allow some leeway while maintaining good coverage practices.

Integrate Code Coverage Into Code Reviews

Incorporating code coverage checks into the code review process encourages teams to monitor testing quality actively. One best practice is to add coverage reports to every pull request so it can be reviewed along with other aspects of the new code (Codacy supports this feature). 

This fosters a culture of accountability, ensuring that testing standards are upheld across the team, and helps detect missing or inadequate tests early in the development cycle.

Understand That Good Coverage Does Not Equal Good Testing

A strong testing culture isn't just about having high code coverage; it’s about ensuring your team understands how the application should behave under normal and exceptional conditions. Coverage tools can help identify areas that require testing, but they won’t tell you if your tests are robust enough to handle unexpected scenarios.

Tests should focus on how the application behaves from a user’s perspective, including edge cases and unexpected inputs, not just hitting every line of code.

Strive for high coverage, but don’t neglect the importance of comprehensive tests that check the functionality of individual components and the overall system integrity.

Analyze Coverage Reports Regularly

Code coverage should be viewed as an ongoing process rather than a one-time check. Regularly reviewing coverage reports helps you track progress, identify test gaps, and detect potential issues early. 

Over time, coverage trends can highlight areas that require more attention or testing strategies that are no longer effective. Constantly refining your tests and test cases is key to keeping coverage relevant and aligned with the evolving codebase.

Use Mutation Testing To Evaluate The Efficacy of Tests

Mutation testing helps you evaluate the quality and robustness of your tests. By introducing small, controlled changes (mutations) to your code and checking whether your tests detect these changes, you can assess how well your tests catch faults. 

Tools like Stryker (for JavaScript, Java, and .NET) or Pitest (for Java) are useful for this purpose, offering valuable insights into areas where your tests may be weak or incomplete.

Continuously Improve Coverage Strategy

Code coverage should not be static; your strategy should evolve alongside the project. As new features are added or bugs are fixed, ensure that new tests are created to cover these changes. 

In response to shifts in project scope or complexity, revisit your coverage goals and thresholds. An evolving test coverage strategy ensures that the tests remain effective, relevant, and aligned with the changing needs of the application.

Code Coverage vs. Test Coverage

"Code coverage" and "test coverage" are terms often used interchangeably but have different meanings and outcomes.  Code coverage is about the amount of code executed. It tells your team what areas of code have and haven’t been executed, while test coverage informs your team of the risks that have been evaluated and examined. 

Code coverage encompasses various metrics measuring the extent to which a test suite executes your source code. It provides information about which lines, branches, conditions, or functions in your code have been executed during testing. 

It’s important to remember that code coverage is not limited to just tests; it includes any method of code execution, such as manual testing or other forms of testing like static code analysis.

Test coverage refers explicitly to the proportion of your codebase exercised by a set of tests. It quantifies how well your tests exercise different parts of your code. Therefore, test coverage is a component of code coverage, providing a more focused perspective on the effectiveness of your test suite in testing the codebase. 

Test coverage can be achieved through various testing techniques, each serving a specific purpose in ensuring the quality and reliability of software:

• Unit tests involve testing individual units or components of a software application in isolation. Units are the smallest testable parts, such as functions or methods. 

• Responsive tests focus on validating that a web application or website displays correctly and functions well across different devices and screen sizes.

• Cross-browser tests verify that web applications or websites work across browsers to ensure compatibility and consistent behavior.

• Integration tests verify interactions between various components or modules of a system.

• Acceptance tests evaluate whether a software application meets specified requirements and is ready to deploy. 

• Regression tests ensure that new code changes do not negatively impact existing functionality.

Development teams measure test coverage to ensure critical business requirements are prioritized via testing for a variety of coverage types: 

• Requirement coverage ensures that the testing process aligns with the specified functional and non-functional requirements, validating that the software meets its intended purpose.

• Risk coverage directs testing efforts toward areas of the codebase that pose potential risks, helping teams prioritize testing activities and mitigate critical issues.

• Product coverage thoroughly evaluates the entire software product, ensuring that different features and components are adequately tested.

By considering these aspects in addition to traditional metrics like line, branch, and statement coverage, development teams can adopt a more thorough and nuanced approach to test coverage, addressing different dimensions of software quality and reliability.

Code Coverage Tools

No matter what your team’s preferred programming language is, plenty of code coverage tools are available to help you gather data on the tests you’re running, analyze them, and generate coverage reports. 

Once you run and execute your tests, these tools calculate the percentage of code executed, after which they generate a code coverage analysis report.

For quick reference, here is a list of popular code coverage tools by programming languages that are worth checking out:

• Java: JaCoCo, Cobertura
• Javascript: Istanbul, jscoverage
  
• Ruby: SimpleCov, undercover
  
• Python: pytest-cov, Coverage.py
  
• C#: Coverlet, NCover
  
• C++: Parasoft Jtest, Testwell CTC++
  
• Rust: Tarpaulin, grcov 
  
• PHP: Xdebug, PHPUnit
  
• Dart: test-coverage, coverage
  
• Scala: scoverage, scct

Centralize Your Code Coverage Insights 

Once you’ve picked the coverage tools that work best for your team, the next step is finding a way to monitor your code coverage reports easily. This can be especially daunting for large software companies with many repositories to monitor. 

Codacy Coverage makes this easy, providing a repository dashboard with a simple and effective way to monitor all your coverage. All you need to do is generate coverage reports for commits on your CI/CD workflow and then upload the coverage data to Codacy.

The dashboard provides valuable data at a glance, including a code coverage percentage, coverage evolution reports, and a list of open pull requests for every repository.

Start a free trial today to get a closer look at how Codacy’s coverage dashboard works and how easy analyzing and reporting on code coverage can truly be.