Home Code Coverage Take your code coverage to the next level

Take your code coverage to the next level

Author

Date

Category

On October 25th, we did a webinar called Take your code coverage to the next level. A panel of Codacy engineers, managers, and sales leads discussed coverage metrics, lessons learned from tracking our code coverage at Codacy, dealing with legacy code, and much more!

Whether or not you are using our Codacy Quality product to generate coverage reports, in this live talk you can learn:

  • The importance of code coverage and how it impacts code quality;
  • What are some different ways to measure code coverage;
  • How we track line coverage at Codacy across different teams;
  • How to manage legacy code and guarantee coverage;
  • Building an improved coverage for our product.

In case you missed the talk live, don’t worry: you can (re)watch the recording here or below 👇

A live talk on code coverage

You can check the detailed talk on our video recording – we even give you the specific time frames! But we’ve summarized the topics for you to read 🤓

Code coverage comics

Why you should care about code coverage (00:04:14)

Let’s get clear: no codebase is perfect. As such, testing is a critical part of the software development process. When talking about code coverage, we refer to the percentage of code covered by tests.

You and your team should prioritize code coverage as it helps maintain and improve your overall code quality. Code coverage allows you to, among others:

  • Have visibility on the state of your code;
  • Maintain test quality over a project’s lifecycle;
  • Inform technology audits and due diligence (e.g., by investors);
  • Promote a culture of writing unit tests and setting standards;
  • Increase confidence when changing code;
  • Improve user experience and retention.

Finally, we know that the market has been evolving in recent years regarding code coverage. While test reports help pass new features through QA, there are new ways of leveraging your coverage metrics:

  • Coverage visibility – Get visibility of code coverage across all your projects and files;
  • Coverage goals – Set code coverage goals across your repositories;
  • Coverage gates – Enforce code coverage by establishing gates for your Pull Request.
Code coverage 2.0

How Codacy’s back-end engineers use coverage (00:08:47)

Codacy has been around for more than 8 years, and some parts of our codebase were not well covered. However, many teams are using our products, so we need to keep growing and adding more features. We can’t stop everything and focus only on improving our coverage.

Our back-end engineering team strives for better overall coverage, making them more confident that new changes won’t break existing behavior. The strategy is to improve while they grow the product and change the code. In a nutshell, the team tests both new code and code that solved an issue, and changes to legacy code should also result in further tests.

Why do we use coverage

The team can access a coverage report in their testing environment, but they need more. As such, the back-end engineering team uses Codacy Quality to:

  • Track the evolution of coverage over time
    • Repository-level metrics allow exploring total coverage over time at set goals
    • There is a visual trend in the coverage evolution
  • Enforce proper coverage on code changes
    • Rely on a metric that covers well incremental changes: diff coverage gives information about code coverage but scoped to changes in the PR or Commit
    • Diff coverage is enforced with Quality Gates so that you can only merge code that meets an acceptable diff coverage percentage (the team uses 60-70%)
  • Prevent degradation of coverage
    • Enforcing that a PR may not drop the total coverage for the repository

How Codacy’s front-end engineers use coverage (00:18:11)

The way Codacy’s front-end team uses coverage has points of contact with the approach followed by the back-end team. However, this team mainly focuses on coverage variation. 

This team mainly works on two projects: one is a component library, and the other is the UI itself. These two projects are different in terms of what can be tested. Since the information on code coverage is over unit tests, the team understood that, with a component library, they could reach higher coverage values than on the UI, where they test some of the logic with integration or end-to-end tests.

Still, the team wanted to write more unit tests. For them, the way to achieve this was to set a quality gate that says their coverage variation needs to be higher than -1% for the code to be accepted.

Coverage variation - quality gate

The team doesn’t check the variation for every single Pull Request. However, if the PR degrades the overall coverage value, they can control it and force them to write more tests.

Finally, the evolution of the coverage at the repository level is something that this team values the most because it allows them to see the overall trend. That way, they can make sure coverage is not decreasing.

Coverage variation

Learning from the best: VEVO case study (00:23:40)

Vevo was founded in 2009 and is a leading music network. When they came to Codacy, they sought a solution to help them with a 12-year-old codebase with low coverage. The goal was to tackle legacy code to move faster, build a continuous delivery pipeline, and measure quality and coverage.

Their approach was simple but efficient: to set up PR quality gates where every team could select what level of coverage they find feasible to achieve. Then, over time, those teams could ramp up that threshold. Ultimately, they went towards 70-80% diff coverage for new code on every PR.

As a result, Vevo reduced their tech support time by about 60% for their YouTube publishing pipeline. Read more on how Vevo uses Codacy to replace legacy systems while guaranteeing code coverage here.

Our code coverage path in 2022 and what we planned for 2023 (00:26:17)

During 2022, we focused on a mix between increased reliability and new features. For coverage, we implemented a new diff coverage metric and added decimal places for coverage variation to give you better insights.

At Codacy, this year alone, we saw a 50% increase in user demand (mostly organic) and a 20% increase in sales requests related to code coverage automation. So this is becoming an increasingly important topic in the industry. As you can see, code coverage is the way to go.

Product perspective - Our coverage path in 2022

We already have ideas for 2023, but we would love to have your input! Feel free to submit your idea to our product roadmap. We plan to have a dedicated team working on coverage, so we’ll have exciting news to share soon.

Product perspective - What would you like to see in 2023?

Q&A time

After the talk, we opened the floor to all the questions the audience might have. We were delighted to receive a tone of questions! We’ve listed them for you: 

Thank you to everyone who joined live and those who watched the recording later on! Let’s keep the conversation going in our community.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Subscribe to our newsletter

To be updated with all the latest news, offers and special announcements.

Recent posts

Measuring and improving software development productivity

Over the years, companies have followed different approaches to measure software development teams’ productivity. The choice was vast, from lines of...

3 popular Python style guides that will help your team write better code

A code style guide is a set of rules, standards, or best practices that outline how your team should write, format,...

Great news: Unity support is now live!

You spoke; we listened! So, finally, after some wait, we’re excited to announce that Codacy Quality now supports Unity through client-side...

October Product Update 🚀

Hi there 👋, Here are a few updates from October that we've been working on to improve...

Take your code coverage to the next level

On October 25th, we did a webinar called Take your code coverage to the next level. A panel of Codacy engineers,...