How to Refactor Your Legacy Codebase

The phrase “legacy codebase” conjures up images of decades-old COBOL code that only one old-timer understands.

But that isn’t what legacy codebases are. Codebases can become “legacy” very quickly. It might just take a few months of non-maintenance for a codebase to go from active and vibrant to stagnant and legacy. When that happens, getting the codebase on track is more complex than if maintenance had continued.

Transforming a legacy codebase into something that is maintainable, scalable, and adaptable to new requirements is challenging but crucial for ensuring the long-term success and viability of the software system. Here, we want to look at shifting a legacy codebase to a healthy one, making it more efficient and secure.

How Codebases Become Legacy

Before transforming legacy codebases, let’s see if any of these ring a bell. Codebases can become legacy due to various factors.

The first is technological obsolescence. Legacy codebases often rely on outdated technologies, frameworks, or libraries that lack modern features, performance improvements, and security updates. 

These outdated components may no longer be supported by their original developers, making it challenging to address bugs, vulnerabilities, or compatibility issues. Furthermore, the lack of modern features and optimizations can hinder the codebase's ability to meet business requirements and performance expectations, affecting the software's competitiveness and user experience.

A second way is an accumulation of technical debt. Short-term solutions, workarounds, and suboptimal design choices accumulate over time, creating technical debt that hinders the codebase's maintainability and scalability. Even if the original design was good, quick fixes, patches, and ad-hoc changes could erode the codebase's original architecture, making it harder to understand, maintain, and extend.

The third way is more human. As developers leave the project or organization, crucial knowledge about the codebase's design, quirks, and undocumented features can be lost, making maintenance and modifications more challenging.

Or humans don’t update documentation. Insufficient or outdated documentation hinders new developers from understanding the codebase's structure, dependencies, and functionality, slowing development and increasing the risk of introducing bugs.

The Challenges of Updating Legacy Codebases

Challenge #1: Dependency Hell

Legacy codebases often suffer from complex and outdated dependencies that are difficult to manage and upgrade. Over time, these dependencies can become obsolete, unsupported, or incompatible with newer code versions.

Updating a single component in a legacy codebase can trigger a cascading effect, requiring updates to a chain of dependencies. This is known as "dependency hell," where upgrading one library necessitates updating multiple other libraries that depend on it. The process can be time-consuming and risky, as each update may introduce bugs, break existing functionality, or require significant code changes to maintain compatibility.

For example, consider a legacy Java application that uses an outdated version of the Spring framework, which depends on an older version of the Apache Commons library. Updating the Spring framework to a newer version may also require updating the Apache Commons library. However, the newer version of Apache Commons may have breaking changes or incompatibilities with other parts of the codebase, leading to a chain of necessary updates and potential conflicts.

This was the problem Stim faced when updating their legacy codebase. In their case, it was really “legacy.” Stim has some ancient systems still running, including an application deployed in 1986 and written in the RPG programming language. These older systems, based on IBM technologies like RPG and Lex, needed to be maintained.

"So as you can imagine, we have a lot of legacy code, a lot of old tech that are still running very well but needs to be maintained. So there's a set of unique challenges around what we are doing," said Tobias Sjösten, Head of Software Engineering at Stim.

Resolving dependency issues in legacy codebases often involves careful analysis of the dependency graph, assessing the impact of updates, and potentially rewriting or adapting code to work with newer dependencies. This process can be complicated by transitive dependencies (dependencies of dependencies), conflicting version requirements, and a lack of clear documentation or support for older versions.

This is where Codacy Quality helped Stim. By analyzing their codebase, Stim was able to identify problem areas of code. Before adopting Codacy, Stim had little visibility into the quality of its legacy codebase, which spanned 190 repositories. Codacy provided comprehensive reporting to surface critical issues, test coverage gaps, etc. This allowed them to prioritize refactoring efforts.

For this legacy application that was too daunting to modify, Codacy enabled bite-sized refactoring by surfacing minor issues developers could quickly fix. This gave teams increasing ownership over legacy codebases.

Challenge #2: Low Coverage

Legacy codebases often suffer from low test coverage. Comprehensive test coverage is essential for ensuring a codebase's reliability, maintainability, and stability. However, many legacy systems were developed at a time when testing practices were less mature or prioritized, resulting in inadequate or missing tests.

The lack of comprehensive test coverage in legacy codebases poses several technical challenges:

1. Risk of introducing bugs: Without a reliable test suite, developers cannot verify that changes or refactorings do not introduce new bugs. When modifying the codebase, developers rely on tests to catch unintended side effects or regressions. If the test coverage is low, there is a higher risk that bugs may slip through undetected, leading to production issues and customer impact.
   
   
2. Difficulty in identifying breaking changes: Tests serve as a safety net to identify breaking changes when modifying code. If a change breaks existing functionality, a well-designed test suite should fail, alerting developers to the issue. However, with low test coverage, breaking changes may go unnoticed, leading to unexpected behavior or failures in production.
   
   
3. Increased manual testing effort: When automated test coverage is lacking, teams often use manual testing to ensure the system's correctness. Manual testing is time-consuming, error-prone, and less efficient than automated tests. It requires significant effort to thoroughly test all possible scenarios and edge cases, slowing the development and release process.
   
   
4. Lack of confidence in refactoring: Refactoring is essential for improving code quality, maintainability, and performance. However, without comprehensive test coverage, developers may hesitate to perform significant refactorings as they lack confidence that the changes won't introduce new bugs. The fear of breaking functionality hinders improving the codebase over time.

Stim faced all of these challenges. Before adopting Codacy, Stim had very low test coverage across their codebases. “I think we had around 10 repositories which were tested before, so it was a quite small percentage, and the average of those repositories were about 19%," said Tobias.

To address the challenge of low test coverage in legacy codebases, organizations can adopt the following strategies:

1. Gradual test coverage improvement: Incrementally add tests to the codebase, focusing on critical paths, high-risk areas, and frequently modified code. By prioritizing the most essential parts of the system, teams can gradually increase test coverage over time.
   
   
2. Test-driven development (TDD): Encourage developers to write tests before writing the code. TDD ensures that new code is developed with testability in mind and helps improve overall test coverage.
3. Legacy code refactoring techniques: Use characterization tests, golden master testing, and seams to safely refactor legacy code and introduce tests. These techniques help understand and document the code's existing behavior, making adding tests and performing refactorings easier.
   
   
4. Automated testing frameworks: Adopt modern testing frameworks and tools that support the legacy codebase's programming language and technology stack. These frameworks provide a structured way to write and run tests, making creating and maintaining a comprehensive test suite more accessible.
   
   
5. Continuous integration and continuous testing: Use continuous integration (CI) practices to integrate automated tests into the development workflow. Ensure that tests are run automatically on each code change, providing early feedback on potential issues and regressions.

Stim set up quality gates in their development workflow using Codacy, such as requiring 80% test coverage on new code. This drove incremental quality improvements, taking average coverage from around 20% to over 60% in 70 repositories within nine months. 

Organizations can mitigate the risks associated with low test coverage in legacy codebases by gradually increasing test coverage, adopting modern testing practices, and leveraging automated frameworks. Improved test coverage enables developers to make changes confidently, catch bugs early, and ensure the long-term maintainability and reliability of the system.

Challenge #3: Security Issues

Legacy codebases often contain outdated components, libraries, and frameworks that may have known security vulnerabilities. Identifying and addressing security issues in legacy systems is crucial to ensuring the organization's overall security posture. 

Legacy codebases may also have been developed when secure coding practices were less mature or not widely adopted. This can result in common security vulnerabilities like SQL injection, cross-site scripting (XSS), or insecure deserialization. These vulnerabilities can be deeply ingrained in the codebase and may require extensive code analysis and refactoring.

They often also lack comprehensive security testing, such as penetration testing, vulnerability scanning, or static code analysis. Without regular security assessments, vulnerabilities may go undetected for extended periods, increasing the risk of exploitation. Integrating security testing into the development lifecycle of a legacy codebase can be challenging, as it requires specialized skills and tools.

When Stim first started using Codacy, one of its applications had 55,000 critical issues flagged, including potential security vulnerabilities. This highlights the accumulation of technical debt over time.

"I have to say, but it wasn't without pain, though, because seeing those, we had one application that, when we started tracking it, I think it had 55,000 critical issues in it. So all from SQL injections to weird execution of dynamic code and so on," Tobias said.

To address security issues in legacy codebases, organizations can adopt the following strategies:

1. Vulnerability assessment: Conduct regular vulnerability assessments and penetration testing to identify security weaknesses in the legacy system. Automated vulnerability scanning tools and manual testing techniques uncover potential attack vectors and prioritize the most critical vulnerabilities for remediation.
   
   
2. Dependency management: Implement a robust dependency management process to track and update outdated libraries and components. Regularly monitor for security advisories and patches related to the dependencies used in the legacy codebase. Assess the impact and feasibility of updating dependencies, considering compatibility, performance, and stability factors.
   
   
3. Secure coding practices: Educate developers on secure coding practices and provide training on identifying and mitigating common security vulnerabilities. According to our 2024 State of Software Quality report, 46% of development leaders provide ongoing security training for developers. Implement secure coding guidelines and standards specific to the programming language and framework used in the legacy codebase. Use static code analysis tools to automatically detect potential security issues in the code.

Implementing Codacy With Your Legacy Codebase

Like Stim, you can use Codacy to help transform a legacy codebase.

• Codacy Quality can help you identify and prioritize technical debt, complex code, duplication, and other maintainability issues in your legacy codebase, enabling you to make informed decisions about refactoring and improvements.
  
  
• Codacy Coverage can help you assess the test coverage of your legacy codebase, identify gaps in your test suite, and set quality gates to enforce higher coverage levels, thereby reducing the risk of introducing bugs and improving overall code reliability.
  
  
• Codacy Security can help you detect security vulnerabilities, insecure coding patterns, and outdated dependencies with known security issues in your legacy codebase, providing actionable insights to remediate these risks and improve the overall security posture of your application.

Let’s take a legacy codebase and see what Codacy suggests. First, we’ll look at the quality of the codebase. To start with Codacy Quality, you just need to sign up for Codacy for free and add the code repository.

Codacy will then scan the entire codebase, looking for quality issues:

In this case, we’re lucky that the codebase doesn’t have many errors, and most issues are related to code style. However, we do have some security issues that need to be assessed. These security issues have gone unaddressed in this legacy codebase, presumed for years before being surfaced by Codacy Quality.

A quick look at the specific issues tells us what the main problems are:

Let’s start quickly with the requests timeout problem. Not adding a timeout can cause issues with responsiveness. From a security perspective, not having a timeout opens you up to DDoS attacks, where an attacker can open multiple connections that won’t timeout and overload your resources.

The more common security problem we see in this codebase is the use of pickle. The Python pickle module is a popular serialization library that allows you to convert Python objects into a byte stream and vice versa. However, using pickle for deserializing untrusted data can lead to serious security vulnerabilities. Attackers can craft malicious pickle payloads that execute arbitrary code during the deserialization process, potentially compromising the security of your application and the underlying system.

We now have two security issues to add PRs for and need to clean up the code styling to fit PEP 8.

We can also see that there are a few open PRs currently that raise concerns:

When we look at these, we can see that most issues are again with styling, with some possible runtime errors as well. But a bigger problem for which we would want to use Codacy Coverage is the third PR on the list—”Add first tests.”

Tests haven’t been added to the codebase if that PR is open. Thus, coverage is currently at 0%. The first step here is to add Codacy Coverage. We can do that by first adding our API token to our project:

export CODACY_PROJECT_TOKEN=<CODACY_API_TOKEN>

Then, we can add the following line to our CI/CD pipeline so that coverage reports are uploaded to Codacy:

bash <(curl -Ls https://coverage.codacy.com/get.sh)

Then, as we add tests, those reports will go through to our Codacy dashboard so we can see the coverage.

Finally, let’s look at security. Here’s our report:

There’s pickle again. But, from a security perspective, we have at least two other issues:

1. The application uses hardcoded passwords, which are critical security vulnerabilities because they allow unauthorized access if the password is discovered.
   
   
2. Open redirect vulnerabilities were found due to the application allowing arbitrary URLs in 'urllib' methods, potentially enabling attackers to craft malicious links.

Both of these are problems, with one requiring a code fix and the other requiring a rethink about the modules included. But both wouldn’t have surfaced without Codacy analyzing the codebase.

Lots to do to transform this legacy code into clean, secure, high-quality (and tested) code.

The Human Element

Before adopting Codacy, stim had little visibility into the quality of its legacy codebase, which spans 190 repositories. Codacy provided comprehensive reporting to surface critical issues, test coverage gaps, and other issues, allowing stim to prioritize refactoring efforts.

You can do the same. Adding Codacy to your projects allows you to view your entire codebase for quality, coverage, security issues, and more. 

But Stim highlighted another issue: people. Finding these problems is good, but your developers still need to fix them. And when you get a report that a legacy codebase needs hundreds of fixes, that can be daunting.

As Tobias said, "We've had some applications that were this type that no one wanted to touch. Whenever you got a Jira ticket assigned to you to fix this old system, you're like, okay, I'll quit my job rather than fix this up."

However, by using Codacy to identify bite-sized issues and dedicating time to regularly fix them, they were able to gradually improve these codebases and change the team's attitude:

"But having these processes in place where you get a bite-size problem, like fix this problem, fix this formatting, or fix this, it adds a test to just check this thing. That grew some responsibility and ownership within the teams,” added Tobias.

“So that was very interesting to see how that changed teams from being not wanting to touch these repositories with a stick, to being like, these are our repositories, they're really good, we are proud of them."

This is what we want at Codacy–for your teams to understand that quality, security, and coverage aren’t unnavigable issues but problems that can be addressed together to shift a legacy codebase into performant, modern code. You can sign up for Codacy today to start transforming your legacy codebase as well.