Should an AI agent fix your vulnerable dependencies? When to let it, when to gate it

In this article:
Subscribe to our blog:

Bots and coding agents now open dependency pull requests at volume, and engineering teams have to decide which of them merge without a person reading them.

This article looks at which dependency fixes an agent can merge on its own, using VulnBump, a benchmark of agents fixing vulnerable dependencies, and our re-read of its per-attempt results. We then turn the evidence into a merge policy by update type, with the checks to require on every dependency pull request.

TL;DR: Merging AI dependency vulnerability fixes

  • Auto-merge only patch updates to direct dependencies at 1.0.0 or later that change just manifests and lockfiles. Require passing checks and a 7-day wait after release. Security patches skip the wait, not the checks.
  • Gate everything else for a person: minor, major, and 0.x updates, transitive fixes, and any agent pull request that edits application code or tests.
  • The evidence comes from VulnBump data (307 upgrade tasks from 43 real repositories), an independent agent study by Luo et al., and peer-reviewed research on how libraries and tests behave.
  • Treat a green build and the agent's own report as weak proof. Require a strict install, a rescan, the build, the tests, and static analysis on the changed code.

Which dependency fixes can an agent merge without a person looking?

Patch updates to direct dependencies are the fixes an agent can merge on its own, provided independent checks pass. Everything else should wait for a person.

The common assumption is that an agent that fixes a CVE has finished the job. The data points the other way. The agent's fix is a change like any other and needs the same gate, because agents can report "fixed" when the vulnerability remains, and a green build says little about whether it is gone.

The question matters now because bots and coding agents open dependency pull requests at scale, and engineering teams are deciding which to auto-merge.

Automation is worth having. In a study of 832 Maven CVEs, teams adopted the fixed release a median of 10.4 days after it shipped when using Dependabot version updates, compared with 110.6 days without them (Hasan et al., preprint). Only 8.5% of adoptions had those updates configured.

Getting this rule right is what lets a team move toward an AI-native SDLC, where agents do more of the work, and the gates, not a reviewer reading every line, decide what ships.

Without a clear merge policy, teams either review everything by hand and lose the speed agents offer, or merge too much unchecked and take on risk they can't defend.

How we evaluated whether you should let an AI agent fix your vulnerable dependencies

To reach our conclusions, we used public data, with VulnBump as the main benchmark, and ran our own re-read of VulnBump's per-attempt results.

Source

Type

What it covers

VulnBump

Preprint. The authors are Snyk employees.

307 upgrade tasks from 43 real repositories, 2,066 fixable vulnerability instances, and six package managers (Maven 18 repositories, Gradle 3, npm 11, Yarn 3, pip 3, Poetry 5).

VulnBump's authors scanned the repositories on 4 to 5 October 2026.

All agents ran Claude Sonnet 5.5, with one attempt per upgrade.

Source: the VulnBump paper.

Our re-read of VulnBump

Our own analysis, 5 November 2026

The per-attempt results published in the VulnBump repository (Apache 2.0).

We first reproduced VulnBump's headline pass counts and update-type split. Then we counted results for the groups the policy treats differently: patch updates to direct dependencies at version 1.0.0 or later, transitive fixes and major upgrades, with side effects in each.

The steps are in "Method and data" at the end.

DepBench

Preprint, 31 August 2026

Independent agent evidence by Luo et al., from researchers at Monash University, Peking University and Microsoft, covering current models from three vendors.

We cite it as Luo et al. because an unrelated benchmark shares the name.

Library and test studies

Peer-reviewed, 2022 and 2023

Structural evidence on how libraries and tests behave, with dates in the text.

Other sources and notes:

  • Hasan et al. and Tanaka et al. are also preprints.
  • Claims about what agents can do use 2026 sources only.
  • VulnBump and DepBench numbers never appear in the same comparison because their tasks differ.

Limits of our analysis. The auto-merge group is 37 upgrades from 14 repositories. 18 of the 37 come from one repository, a Java tutorial project with many small sample builds, and 24 of the 37 are Gradle. Outside Gradle and Maven, the samples are tiny. The data has no field for edits to application code, so it cannot test that rule.

Note on VulnBump limitations: Snyk built the tasks, and its scanner is the judge. New vulnerabilities and unrelated dependency bumps do not count against an agent's pass. It covers six package managers, with one run per upgrade.

The repositories are pinned, but the vulnerability database is not, so a rerun today finds more than 2,066 instances and may recommend different versions.

VulnBump results: four ways to fix vulnerable dependencies

VulnBump compared four ways of fixing the same 307 vulnerable upgrades, from a plain version bump to a purpose-built remediation agent.

The table shows how often each passed all four gates and what share of vulnerability instances it fixed.

Approach

What it is

Passed all 4 gates

Vulnerability instances fixed

Package manager bump

No agent. Applies the recommended version only. Closest to a plain Dependabot or Renovate bump.

46.9% (144/307)

56.5%

Claude Code unaided

General-purpose agent, no target version, no scanner access

59.6% (183)

72.7%

Claude Code with scanner data

Same agent given the target version and CVEs

77.5% (238)

83.1%

Evo Remediation Agent

Purpose-built remediation agent

90.2% (277)

95.9%

An upgrade counted as a pass only if it cleared four gates:

  • The dependencies had to install with the same strict install command your CI uses, which fails if the lockfile and manifest disagree.
  • A rescan had to show the targeted vulnerabilities gone.
  • The project had to build.
  • Every test that passed before had to pass again, with a skipped or missing test counted as a failure.

Against those gates, scanner data mattered more than which agent ran. When Claude Code was given the target version and the CVEs, it passed 77.5% of upgrades (238 of 307). Without them, it passed 59.6% (183), and it picked a version that was still vulnerable 64 times, against 13 with the data.

Without scanner data, the agent had to find a fixed version by searching the web, and VulnBump reports that this "proved unreliable".

Transitive fixes were the hardest. These are vulnerabilities in a dependency of a dependency, so your own manifest doesn't name them.

They make up 70% of fixable instances, and every setup fixed them less often than direct ones:

  • Evo 73% against 91%
  • The agent with scanner data 59% against 71%
  • The unaided agent 28% against 62%

Why isn't a green build proof that the vulnerability is gone?

A green build only shows the code compiles and the tests pass. In VulnBump, every agent kept the build green about 93% of the time, but removing the vulnerability was the hard part.

The table counts the side effects VulnBump recorded per attempt, across all 307 upgrades.

Signal (attempts)

Claude Code unaided

Claude Code with scanner

Evo

Claimed a fix the rescan still flagged

70

2

1

Introduced new vulnerabilities

33

30

62

Tests modified

22

26

51

Unrelated dependency bumps

34

26

74

 

We identified three main problems:

  • The first problem is that an agent's own report can't stand in for a check. The unaided agent claimed a fix that the rescan still flagged in 70 attempts. We checked the same thing on a smaller set: the 37 upgrades our policy would auto-merge.

    There, the unaided agent made 10 false fix claims, and the agent with scanner data made none. This supports two rules: give the agent scanner data, and rescan instead of trusting its report.
  • The second problem is that a fix can pass inside the agent's session and fail everywhere else. VulnBump found lockfiles out of sync and an unsaved --legacy-peer-deps flag. Of the 30 attempts that failed the strict install, 19 passed every other gate.

    A strict install run the way your CI runs it catches that class of failure, and a build run in the agent's own environment does not.
  • A third problem is that a passing upgrade can still pull in new vulnerabilities. Because VulnBump's pass doesn't penalize new vulnerabilities, post-fix scans found new ones in 33 unaided attempts, 30 with scanner data, and 62 for Evo.

    Agents sometimes bumped dependencies unrelated to the fix (34, 26, and 74 attempts, in the same order), which is why a person should see the full diff before merge.

Independent research points the same way: passing tests is a thin check.

In DepBench, 322 of 521 analyzed failures passed the project's visible tests before failing the hidden tests taken from the real human fix. The most common cause was an incomplete migration, where the agent found the right change and missed call sites.

This is why the merge policy we propose never treats a green build as the verdict. It asks for evidence the agent doesn't control: an independent rescan that shows the vulnerability gone, a strict install run the way your CI runs it, and a person to review anything other than a patch update that changes only manifests and lockfiles.

The merge policy: which updates to auto-merge and which to gate

The policy sorts every dependency fix by update type. One group an agent can merge alone. The other four wait for a person.

Patch updates: let the agent auto-merge them

Auto-merge patch updates to direct dependencies at version 1.0.0 or later when the change touches only manifests and lockfiles. Merge them after the required checks pass, and a 7-day release delay has run. Security-driven patches skip the delay, and every check still applies.

Across all 57 patch upgrades in VulnBump, the agent with scanner data passed 98.2% and Evo 96.5%, against 77.2% for the unaided agent (VulnBump results, "Version jump").

Our cut of the 37 upgrades the policy would auto-merge shows the same. The agent with scanner data and Evo passed all 37, with no new vulnerabilities and no unrelated dependency changes.

Security-driven patches skip the delay because the version you run is already known to be vulnerable. The rescan, the strict install, and malicious-package checks on the pull request still run.

Minor updates: gate them for a quick human review

Minor updates should open a pull request, get a label, and wait for human approval. The agent is still capable here, but the numbers move.

With scanner data, the agent's pass rate drops from 98.2% on patch to 86.2% across 123 minor upgrades.

Ochoa et al. found binary breaking changes in 37.9% of minor upgrades in their 2022 Maven study. That is why the line sits here. The agent usually gets it right, and the structural risk is real enough that a person should look.

The review is quick and bounded: check the changelog, check what the agent touched, approve.

Major upgrades and 0.x versions need human review

Major upgrades should be gated with a full human review, and any 0.x version should be treated like a major. Semantic versioning gives 0.x releases no compatibility promise.

Across 118 major upgrades, the agent with scanner data passed 61.0%, Evo 80.5%, the unaided agent 42.4%, and a plain bump 25.4%. Twenty-three major upgrades, mostly Spring, Spring Boot, and Dropwizard, defeated every agent.

Minor upgrades on 0.x behaved like majors, at 44.4% with scanner data, though the sample is only 9.

Side effects grow with the jump. In our cut, the agent with scanner data introduced new vulnerabilities in 25 of 118 major attempts and made unrelated dependency changes in 20. In the auto-merge group, both counts were zero.

VulnBump's own authors agree with the gate. The VulnBump paper says major migrations "may still warrant human review", and adds that smaller jumps are not always safe, so every fix should be checked against the build and tests, whatever the jump.

In DepBench, the best setup solved 104 of 203 (51.2%) real breaking upgrades that a bot bump could not finish (Luo et al., preprint). The same model differed by up to 33 tasks, or 16.3 points, depending on the harness around it.

Transitive dependency fixes: gate them

Transitive fixes should wait for a person. They make up 70% of fixable instances in VulnBump, so this rule decides most of the volume.

Every VulnBump setup fixed transitive vulnerabilities less often than direct ones, as the figures earlier show.

Tests also reach them poorly: Hejderup and Gousios, in "Can we trust tests to automate dependency updates?" (2022), found tests covered about a fifth of transitive dependency calls and caught about a third of injected transitive faults.

Our cut sharpens this. Even at patch level, every failure by the scanner-assisted agents was a transitive npm dependency. Above patch, the agent with scanner data passed 81.6% of direct upgrades against 64.2% of transitive ones, and the unaided agent 72.8% against 40.0%.

One VulnBump rule is worth copying into your own checks. A vulnerability that now arrives through a new direct dependency counts as "moved", not fixed.

Agent PRs that edit application code or tests: gate them

Any agent pull request that edits application code or tests should get human review, whatever the version jump. A dependency fix that changes source code is a different kind of change and falls outside the patch rule.

VulnBump's gates do not review the code the agent changes, and that gap is visible in the data:

  • Evo edited non-test source in 52% of its passing major upgrades.
  • Tests were modified in 22 to 51 attempts depending on the setup, and modified tests are not proof of tampering.

In DepBench, incomplete migrations were the most common failure (Luo et al., preprint).

Note: Our own analysis cannot test this rule, because the per-attempt data has no field for edits to application source. The rule for any agent fix is simpler still: do not treat the agent's report as evidence. The unaided agent claimed 70 fixes that the rescan still flagged.

A vulnerability remediation process your team can start on Monday

A workable vulnerability remediation process for agent-opened fixes has three parts: a narrow auto-merge rule, required checks on every dependency pull request, and routing for everything else.

  1. Set the patch rule. Auto-merge patch updates to direct dependencies at 1.0.0 or later that change only manifests and lockfiles, after a 7-day release delay. The delay gives maintainers time to catch a malicious release. Among repositories with a general Dependabot cooldown, 64.3% chose 7 days (Tanaka et al., preprint). Security-driven patches skip the delay.
  2. Make the checks required on every dependency pull request, whoever opened it. Require a strict install with the lockfile, a rescan showing the targeted vulnerability gone, the build, and tests where everything that passed before still passes. These are VulnBump's four gates. Add static analysis on the changed code. This covers a gap VulnBump leaves, and Hejderup and Gousios support it.
  3. Route everything else to a person. That covers minor, major and 0.x updates, transitive fixes, and any pull request that edits application code or tests. Cap agent retries at a number your team chooses, then hand the pull request to a person.

The table is the policy in one view, with the strongest evidence for each line.

Update type

Policy

Evidence

Patch, direct, 1.0.0 or later, manifest and lockfile only

Let it, after checks and a 7-day delay

VulnBump 98.2% with scanner data; our cut 37 of 37 (small, concentrated sample)

Minor

Gate it

VulnBump 86.2% with scanner data; Ochoa 37.9% breaking

Major and 0.x

Gate it, with human review

VulnBump 61.0% with scanner data; our cut 25 of 118 attempts introduced new vulnerabilities

Transitive

Gate it

Our cut above patch: 64.2% transitive against 81.6% direct; tests caught 35% of transitive faults

Agent PR editing code or tests

Gate it, with human review

Evo edited source in 52% of passing majors; incomplete migrations most common in DepBench

Sources: VulnBump results; Codacy analysis of VulnBump per-attempt data, run 5 November 2026; Ochoa et al., 2022; Hejderup and Gousios, 2022; Luo et al., 2026 (preprint); Tanaka et al., 2026 (preprint)

Then run the same check on your own history. List the dependency pull requests that auto-merged last quarter, and mark which of them this policy would have gated.

Where Codacy fits: enforcing the gate at the pull request

The policy needs an independent check at the pull request, and that is the part Codacy covers. Codacy does not sell a remediation agent. It checks the fix that a bot or an agent opens, and enforces the result at the PR.

Codacy runs dependency scanning (SCA) on every pull request on its own infrastructure, with merge gates and no CI pipeline to configure.

For a transitive vulnerability, it shows the dependency chain with a prompt to find the affected functions in your code. It checks for malicious packages using the OpenSSF Malicious Packages database.

SAST and code quality analysis run on the same pull request, which covers the static analysis step the policy adds to VulnBump's suggested four gates.

In the Business plan, dependencies are rescanned daily for new CVEs, with remediation deadlines by severity and an SBOM export.

Method and data: how we re-read the VulnBump results

We used the per-attempt results published in the VulnBump repository under benchmarking/reports, for the package manager bump, Claude Code unaided, Claude Code with scanner data, and Evo. The analysis ran on 5 November 2026.

These are the steps, so you can rerun them:

  1. For each setup, take every attempt in cases.*.attempts, skipping cases where baseline.valid is false and attempts where harness_error is true.
  2. Mark an attempt as a pass when its reasons list is empty.
  3. Classify the update type from the versions in label (the version after the last @ on each side of ->) as major, minor, or patch, keeping 0.x versions separate.
  4. Classify the dependency type by looking up each instances entry in the baseline vuln_facts.direct: all true is direct, all false is transitive, anything else is mixed.
  5. Check the reproduction before counting anything new: 307 attempts per setup; passes of 144, 183, 238 and 277 (bump, unaided, scanner data, Evo); 57 patch upgrades including 7 on 0.x, 123 minor, 118 major and 9 minor on 0.x. Our counts matched.
  6. Count passes for the auto-merge group (patch, version 1.0.0 or later, direct), then by dependency type for patch and for all other upgrades.
  7. Count side effects for the auto-merge group and for major upgrades: introduced_count above 0, tests_modified not empty, unexpected_collateral not empty, and claimed_not_fixed true. False claims are ignored for the package manager bump, which is not an agent.

Every number carries the same caveats: a small, concentrated sample, one run per upgrade, Snyk's scanner as the judge, "direct" derived from VulnBump's baseline data, "tests modified" as an observation that does not show tampering, and no field for edits to application source.

Let the agent open the fix. Let the gate decide.

Codacy runs dependency scanning and SAST on every pull request, with merge gates and no CI pipeline to configure. Your patch rule then rests on a check the agent does not control.

Scan your repository for free →

Vulnerability remediation process FAQ

 

What is a vulnerability remediation process?

A vulnerability remediation process is the set of steps that takes a known vulnerable dependency to a verified fix.

For agent-opened pull requests, it has three parts: a narrow auto-merge rule for patch updates, required checks on every dependency pull request, and human review for everything else.

The required checks are a strict install with the lockfile, a rescan showing the targeted vulnerability gone, the build and the tests, plus static analysis on the changed code.

How do I verify an AI agent's fix in a vulnerability remediation process?

Rescan the dependencies on the pull request and run a strict install with the lockfile, because the agent's own report is not evidence. Codacy runs this rescan on every pull request, with merge gates and no CI pipeline to configure.

Should I auto-merge Dependabot or Renovate patch PRs?

Yes, but only for patch updates to direct dependencies at version 1.0.0 or later that change only manifests and lockfiles, once the required checks pass and a release delay has run.

How do I prevent supply chain attacks when bots and agents update dependencies?

Preventing supply chain attacks from automated dependency updates starts with a release delay, which gives maintainers and the community time to detect a malicious release before it spreads.

Among popular GitHub repositories that adopted Dependabot's cooldown, 64.3% of those with a general delay chose 7 days (Tanaka et al., preprint).

The authors describe 7 days as an empirical reference point and do not claim it is optimal. Security-driven patches skip the delay, so malicious-package checks on the pull request still matter.

How do I prioritize fixes in a vulnerability remediation process?

Prioritizing vulnerability remediation means ranking findings so the highest-risk ones are fixed first, and our guide to vulnerability prioritization explains how. When agents open the fixes, the patch auto-merge rule keeps routine bumps away from reviewers, so their time goes to the gated minor, major, 0.x, and transitive fixes.

 

Subscribe to our blog

Stay updated with our monthly newsletter.