Introducing Codacy Skills (Part 3): Let your agent set up test coverage

Codacy tracks four metrics on a repository: issues, duplication, complexity, and coverage.

Three of them we work out by analysing the code. Coverage is the one we cannot, because knowing which lines your tests reach means running the tests, and your tests run in your pipeline rather than ours. So coverage arrives by upload, and it feeds two of the six criteria you can put on a merge gate: Diff Coverage on the lines a pull request changed, and Coverage Variation on a drop in overall coverage.

By hand, that upload is a handful of stack-specific decisions. Getting your test runner to generate a format Codacy accepts, at a known path. Adding the reporter after the test step, with the project token wired in. Making sure it runs on the branch you compare against and not only on pull requests.

The coverage docs walk through all of it, but the answers differ from stack to stack: a Go repo and a Gradle repo, for example, want different report paths, while a monorepo wants a decision about the modules.

The new Setup Coverage Skill wires that up for you.

What is the Setup Coverage Skill?

The Setup Coverage Skill, called with /setup-coverage, sets up test coverage reporting in a repository and wires the upload to Codacy. It reads the repo the way you would if you were setting this up by hand: the languages in use, the test framework, any coverage configuration already present, then the CI pipeline config, to find where the tests run and whether a coverage report is already being sent.

Whatever format your test coverage reports come in and whatever CI system runs them, the skill works out the wiring. It runs in Claude Code, OpenAI Codex, GitHub Copilot and more, through the Agent Skills standard.

It covers eleven language families and seven CI systems, and the skill's own reference has the full list if you want to check yours is in it.

/setup-coverage sets the upload to run on every push rather than only on pull requests. Diff Coverage needs only the commit under review, but Coverage Variation is a comparison, which means that Codacy needs coverage data for the target branch to measure the drop against. The report also covers every tested file rather than only the changed ones, which keeps the overall metric accurate.

Side note: What it will not do is write your tests. If a repo has none it can put the framework in place, but the tests themselves are a separate job, and one worth asking your agent for separately so you can review what it comes up with.

Installing the skill

Installation is unchanged from Part 1 and Part 2, so, if you have those, you have this skill already.

If not, the skills run on the Codacy Cloud CLI. So let's set that up first:

codacy login takes an API token from My Account > Access Management > API Tokens.

Next, install the plugin:

If you use a different AI assistant, visit the Codacy Skills repository to install the skills directly.

Running the skill

Call it with /setup-coverage, or just describe what you want, since the skill triggers on natural language too.

The first pass is read-only. The skill reports what it found, what is missing, and what it proposes to add, before editing anything. That is the useful point to steer it. On a monorepo or a multi-module build there is more than one right answer: you can merge the modules into a single report, or upload them as partials and have Codacy assemble them. Which you want depends on whether you need the modules to read separately.

It also makes the same prompt work as an audit: if coverage is running on some of your repos and not others, point it at one and it will tell you which case you are in.

Then it edits. In a pytest repo, the test command it writes into your pipeline looks like this:

The upload goes in as a step after the tests, using codacy-coverage-reporter-action on GitHub Actions or the reporter script elsewhere. One step stays with you: add the repository's project token to your CI as a secret for the reporter to read. Push, and the next run uploads.

Repository Settings has a Coverage section with a Test your integration panel that shows reports as they land. Give it a couple of commits first, since the pull request metrics need a baseline to compare against.

What's next

With reports arriving, both coverage criteria are available to you, whether you gate on them or just watch the trend. Neither does anything without the data underneath, and the data underneath is now a prompt.

Up next in the Codacy Skills series: the Analysis CLI and the Code Review skill, running Codacy on your own machine before you push the code to Git.