AI Coding Assistant Dependency Version Selection Risks: How Codacy Dogfoods Verity With Its Own AI Agents (2026)

Most teams working with AI coding agents are still at the same stage: a person prompts the agent, reads the diff, and decides whether to ship it. That's real progress, but it isn't a loop yet. Nothing runs on its own, and nothing has to earn a track record before it's trusted with more.

This is the third piece in our loop engineering series. The first one looked at what that shift looks like for a customer, Black Box, running its own pull-request review gates at scale.

In this article, we turn the lens on Codacy itself: how Codacy builds its own review-gate product, Verity, using agents that run inside a loop of their own – one that reviews Verity's own commits before they ship.

How Codacy added autonomy one step at a time

Codacy's agent autonomy grew in deliberate increments rather than a single rollout.

The first move toward anything resembling a loop was modest: point a scheduled job at production logs and tell it to watch and post to Slack, running every hour.

From there, Codacy expanded what the loop could see, starting with read-only access to the production database, the most consequential grant since a compromised or overreaching agent there could touch customer data directly.

The loop's visibility later grew to include Linear and GitHub, but the database permission is the one Codacy checks on every single run rather than assuming it still holds.

The agent learned to score findings for usefulness and group related issues, so it wasn't surfacing the same noise repeatedly, and filed tickets and Slack messages while a person still decided what got accepted.

The next increment let the loop act once a ticket was accepted: write a failing test, implement the fix, open a draft PR. The boundaries stayed concrete. The loop was capped at eight issues and one build per run, and a person remained responsible for accepting, merging, and deploying every change.

Codacy layered in checks around this expansion, including dependency scanning and a Verity review of the diff itself.

When something misfired in production, the fix became a durable rule: a lint rule, an instruction-file line, or a new gate check. Codacy's agent instruction files now carry rules dated to the day something went wrong, preventing the same mistake from resurfacing on its next run.

Overnight self-tuning experiment

For a more ambitious experiment, Codacy’s engineering team wrote a function to score how well a given agent setup performed, passed that score to a fitness function, then let a coding agent run overnight and unattended, adjusting its own model choice and a handful of other configuration settings, and keeping whichever version scored higher on each run.

The first version exposed an obvious flaw: the agent could win by flagging everything, since more flags appeared to be greater diligence to a naive scorer.

Codacy fixed this by grading against known-good and known-bad examples and penalizing false alarms harder than missed bugs. Of everything the loop tried adjusting, swapping the underlying model was the one change that mattered: once it kept that version, false blocks dropped by roughly 75% and the agent caught more real issues in the same pass.

None of it shipped. It stayed an experiment, but one that shaped how Verity was built afterward.

The Four-Level Autonomy Ladder

The four-level autonomy ladder is Codacy's internal model for determining how much unsupervised action a given agent loop has earned, and it describes a loop's track record rather than an industry-wide AI maturity scale.

• Level 1 is "in the loop," where the agent suggests, and a human does the work, which is what Claude Code plus Codacy's own skills look like daily.
• Level 2 is "on the loop," where the agent acts and a human watches and steps in.
• Level 3 is "bounded," where the agent runs unsupervised within a fenced area and asks for help at the edges. It can open a PR on its own, but nothing merges until the automated checks pass and a person approves it.
• Level 4 is "autonomous," running unattended and checked afterward, the territory the overnight fitness-function experiment explored without graduating out of.

The rule governing movement between levels is strict: a loop moves up only after the level below it has been boring for a while.

In this context, boring means no surprises and nothing that required a person to intervene outside the expected pattern.

Autonomy gets earned through a clean track record rather than granted because the technology theoretically supports it, a principle OWASP codifies as Least Agency in its 2026 Agentic Top 10.

What runs day to day: the autobot

The autobot is Codacy's clearest example of Level 2 autonomy in production, an hourly agent that watches production logs and increasingly acts on what it finds, while a person still owns the moments that matter most.

• Read-only access everywhere. It reads production logs, the database, Linear, and GitHub, and the database user carries no write permissions at all, a setting checked on every single run rather than assumed.
• Scores and groups findings. It scores each finding for usefulness and groups related issues together, so a single root cause doesn't become ten separate alerts for the same problem.
• Acts within limits. It posts to Slack, files a Linear ticket, and once a human accepts the ticket, it writes a failing test, implements the fix, and opens a draft PR rather than pushing straight to a branch.

A person stays in exactly three places: accepting the work, merging it, and deploying it.

Nothing skips those checkpoints, and the loop never processes more than eight issues or triggers more than one build per run.

The autobot started as a plain instruction: watch the logs and post to Slack, and grew into its current shape one permission at a time.

Hooks vs. MCP tools: making checks impossible to skip

If a check is optional, the agent skips it, so real enforcement has to be structural rather than requested.

A tool the agent can choose to call is a request the model might honor depending on context and pressure. A hook that fires automatically, independent of what the agent decides, is a guarantee.

In Codacy’s case, two of these checks run mid-session, during the agent's active coding loop, immediately after an edit:

• Static analysis and secrets scanning. Runs after every edit, before the agent moves to the next change.
• Dependency scanning. Runs on every change, using Trivy as the scanning engine, and is invoked through the same codacy_cli_analyze interface used for the static and secrets checks.
  
  This is where AI coding assistant dependency version selection risks get caught directly: an agent that would otherwise pin a stale, bleeding-edge, or non-existent package version gets stopped before that choice reaches a commit.

The rest fire later, once the agent's loop has already produced a change:

• CI integration tests. Run on every push, independent of what the agent believes it already verified.
• A separate review gate. A different model than the one that wrote the code reviews the diff.
• Human approval on a protected branch. The agent cannot merge to main, regardless of what the earlier checks conclude.

Codacy’s team also found a ceiling on how much self-correction helps. Letting an agent retry beyond roughly two rounds usually stopped improving the code and started producing new problems, as the agent over-corrected its own fix and burned tokens chasing a moving target.

Capping self-correction at two passes and handing feedback back to a person proved more productive than trusting the loop to converge on its own.

The other half of this system is self-hardening:

Every repeated mistake becomes a durable rule written into a lint config, an instruction-file line, or a new gate check, so the same failure can't recur silently.

The instruction files Codacy's agents read now carry rules dated to the day something went wrong.

The Guy in the Chair: Ticket-to-PR With Self-Checks

The guy in the chair is a small bot Codacy built entirely inside a normal Claude Code session, with no separate API keys. It does one thing: label a ticket, get back a pull request.

The bot's value comes from refusing to waste a human's attention on ambiguous input before it ever starts building.

The workflow runs in four steps:

1. Grooms first. If a labeled ticket is vague, the bot comments on Jira with exactly what's missing and refuses to build until it's clear.
2. Implements. Once a ticket is ready, it clones the repo, reads the ticket and its full comment history, writes the code, and runs the tests.
3. Checks itself. Before opening anything, it runs Codacy's own checks, including a Trivy scan, documentation checks, and plugin tests.
4. Opens the PR. It commits, opens a GitHub pull request, comments the link back on the Jira ticket, and retries on failure with two attempts by default.

The transferable idea matters more than the specific bot: a system that refuses ambiguous work and checks itself before asking for human attention protects the review gate by not flooding it with half-finished work that someone else has to triage.

Dogfooding: Building Verity With Verity

Verity's review gate runs against Codacy's own pull requests, including recent commits to Verity's own codebase. Trivy scans every dependency bump the same way, before any of it reaches the main branch.

That's the test that matters: would the team that wrote the standard actually ship their own code through it?

A gate a team quietly bypasses for its own commits has already failed that test. Codacy doesn't bypass it. The same checks run on the same codebase that ships the product.

What’s next?

What comes next is agents that earn their way up the ladder described above, one boring track record at a time, plus shared memory so a fleet of agents can work from a single source of truth instead of contradicting each other across repositories.

The more useful exercise for your team is comparative: hold your own setup, its agents, its database permissions, its retry caps, its enforcement model, against the one described here, and find where your review backlog is worst. That's usually where the first gate pays off.