On October 25th, we did a webinar called Take your code coverage to the next level. A panel of Codacy engineers, managers, and sales leads discussed coverage metrics, lessons learned from tracking our code coverage at Codacy, dealing with legacy code, and much more!
Why you should care about code coverage (00:04:14)
Let’s get clear: no codebase is perfect. As such, testing is a critical part of the software development process. When talking about code coverage, we refer to the percentage of code covered by tests.
You and your team should prioritize code coverage as it helps maintain and improve your overall code quality. Code coverage allows you to, among others:
- Have visibility on the state of your code;
- Maintain test quality over a project’s lifecycle;
- Inform technology audits and due diligence (e.g., by investors);
- Promote a culture of writing unit tests and setting standards;
- Increase confidence when changing code;
- Improve user experience and retention.
Finally, we know that the market has been evolving in recent years regarding code coverage. While test reports help pass new features through QA, there are new ways of leveraging your coverage metrics:
- Coverage visibility – Get visibility of code coverage across all your projects and files;
- Coverage goals – Set code coverage goals across your repositories;
- Coverage gates – Enforce code coverage by establishing gates for your Pull Request.
How Codacy’s back-end engineers use coverage (00:08:47)
Codacy has been around for more than 8 years, and some parts of our codebase were not well covered. However, many teams are using our products, so we need to keep growing and adding more features. We can’t stop everything and focus only on improving our coverage.
Our back-end engineering team strives for better overall coverage, making them more confident that new changes won’t break existing behavior. The strategy is to improve while they grow the product and change the code. In a nutshell, the team tests both new code and code that solved an issue, and changes to legacy code should also result in further tests.
The team can access a coverage report in their testing environment, but they need more. As such, the back-end engineering team uses Codacy Quality to:
- Track the evolution of coverage over time
- Repository-level metrics allow exploring total coverage over time at set goals
- There is a visual trend in the coverage evolution
- Enforce proper coverage on code changes
- Rely on a metric that covers well incremental changes: diff coverage gives information about code coverage but scoped to changes in the PR or Commit
- Diff coverage is enforced with Quality Gates so that you can only merge code that meets an acceptable diff coverage percentage (the team uses 60-70%)
- Prevent degradation of coverage
- Enforcing that a PR may not drop the total coverage for the repository
How Codacy’s front-end engineers use coverage (00:18:11)
The way Codacy’s front-end team uses coverage has points of contact with the approach followed by the back-end team. However, this team mainly focuses on coverage variation.
This team mainly works on two projects: one is a component library, and the other is the UI itself. These two projects are different in terms of what can be tested. Since the information on code coverage is over unit tests, the team understood that, with a component library, they could reach higher coverage values than on the UI, where they test some of the logic with integration or end-to-end tests.
Still, the team wanted to write more unit tests. For them, the way to achieve this was to set a quality gate that says their coverage variation needs to be higher than -1% for the code to be accepted.
The team doesn’t check the variation for every single Pull Request. However, if the PR degrades the overall coverage value, they can control it and force them to write more tests.
Finally, the evolution of the coverage at the repository level is something that this team values the most because it allows them to see the overall trend. That way, they can make sure coverage is not decreasing.
Learning from the best: VEVO case study (00:23:40)
Vevo was founded in 2009 and is a leading music network. When they came to Codacy, they sought a solution to help them with a 12-year-old codebase with low coverage. The goal was to tackle legacy code to move faster, build a continuous delivery pipeline, and measure quality and coverage.
Their approach was simple but efficient: to set up PR quality gates where every team could select what level of coverage they find feasible to achieve. Then, over time, those teams could ramp up that threshold. Ultimately, they went towards 70-80% diff coverage for new code on every PR.
As a result, Vevo reduced their tech support time by about 60% for their YouTube publishing pipeline. Read more on how Vevo uses Codacy to replace legacy systems while guaranteeing code coverage here.
Our code coverage path in 2022 and what we planned for 2023 (00:26:17)
During 2022, we focused on a mix between increased reliability and new features. For coverage, we implemented a new diff coverage metric and added decimal places for coverage variation to give you better insights.
At Codacy, this year alone, we saw a 50% increase in user demand (mostly organic) and a 20% increase in sales requests related to code coverage automation. So this is becoming an increasingly important topic in the industry. As you can see, code coverage is the way to go.
We already have ideas for 2023, but we would love to have your input! Feel free to submit your idea to our product roadmap. We plan to have a dedicated team working on coverage, so we’ll have exciting news to share soon.
Thank you to everyone who joined live and those who watched the recording later on! Let’s keep the conversation going in our community.
