How Our CSMs Use Data to Improve Customer ROI

There is an odd thing about “customer success” in most companies. It’s rarely about customer success.

It is mostly about retroactively responding to customer problems and trying to help them fix issues. This is needed, but it isn’t proactively looking for ways customers can be successful.

At Codacy, we’re trying to do things a little differently. Our Data Team has been working with Customer Success to give customer success managers (CSMs) the data they need to help customers proactively.

With the help of internal reporting tools, our CSMs can identify potential pitfalls preemptively, offer tailored solutions based on unique code patterns, and empower our customers with insights, ensuring they achieve optimal code quality even before issues arise.

Here, we want to walk you through this dashboard from the perspective of how our customer success team uses it to help customers get the most out of Codacy and how engineering managers can use this information to lead code quality within their organization.

Our ROI Report Dashboard

Before we get into the “Why” and the “How,” let’s look at the “What.” This is the ROI (return on investment) Report dashboard your Codacy CSM sees when analyzing your repositories:

Here's a breakdown of the various metrics displayed on the dashboard. At the top, we have the Filters. This section allows CSMs to filter the displayed data based on specific criteria:

• Organization: Refers to the specific organization whose repositories are being analyzed.
  
  
• Repo Name: Allows selecting specific repositories to view their code quality metrics.
  
  
• Repo Active: Indicates whether to view metrics for active or all repositories.
  
  
• Timeline: Specifies the period for which the data is displayed (e.g., "Current Week").

After that, we look into Seat Management. Here, we want to see the total number of users in the organization that should be using Codacy. From there, we see the actual number and percentage of total users using Codacy (and the inverse, users who were added but have yet to access the tool).

We want to highlight these metrics as they are essential to the tool's success. If the team isn’t entirely bought into the idea of code quality and using code quality tools, it impacts the rest of the metrics. Finally, we want to look for the number of individuals who have committed code but haven't been added as users in the tool. This is an accurate representation of uptake in your engineering team.

Once we are through these org and team-level metrics, we get to the core metrics of the report.

Repo Quality

Repo quality is a classification of repositories based on their code quality, from Grade A to F. You can see how we determine the repo quality grade in our docs. We then show metrics related to lines of code (LoC) and code coverage in the codebase.

• LOC (Lines of Code) Grade: Indicates the quality of the code based on the number of lines.
  
  
• Active vs. Nonactive: Number of active repositories compared to non-active ones.
  
  
• Avg LoC Covered: A pie chart showing the average percentage of lines of code covered by tests.
  
  
• Files Below Coverage goal: Number of files that don't meet the desired test coverage threshold.
  
  
• Files without Coverage: Files that have no test coverage at all.

Codebase issues

This section provides insights into the various issues detected in the code base.

• Issues Found by Category: Breakdown of issues by category like security, performance, etc.
  
  
• Issues by Severity: A pie chart categorizes problems based on their severity – Critical, Major, and Minor.
  
  
• Issues Summary: A snapshot of the total, new, and fixed issues.

Top Repos

• Top 3 Repos with the highest volume of issues: Displays repositories with the most issues.
  
  
• Top 3 Repos with the lowest volume of issues: Displays repositories with the fewest issues.
  
  
• Top 3 Repos most at Risk, with Security issues: Highlights repositories that are most vulnerable in terms of security.

PRs not up to Standard

This section gives insights about Pull Requests that do not meet certain quality standards.

• Total PRs: The total number of pull requests.
  
  
• PRs Closed but Not Up to Standard: Pull requests that were closed but didn't meet the quality criteria.
  
  
• PRs Open Not Up to Standard: Open pull requests that don't meet the quality criteria.
  
  
• PRs Open Up to Standard: Pull requests that adhere to the set quality standards.
  
  
• PRs Blocked: Pull requests that are blocked for some reason.

Overall, this dashboard offers a comprehensive view of the code quality, which can help teams identify areas of improvement, ensure better code maintainability, and prioritize their efforts effectively.

How Codacy CSMs Use This Data

As a CSM, you aim to ensure the customer derives maximum value from the tool. The idea behind this dashboard is to give CSMs the data they need to help do just that. CSMs can get on calls with customers and walk through this, discussing code quality at the organization and suggesting solutions.

If we take the dashboard above as an example, here is how a CSM might highlight the top priorities and potential solutions.

Repo Quality

We can see two problems here.

First, we have the problem that most repositories are in Grade B (64%), while a significant number are in Grade C (9%). This means that the majority of the code in the codebase isn’t of the highest quality. This is baking in technical debt into your product.

Second, the average Lines of Code (LoC) covered by tests is only 33.56%. This makes the codebase susceptible to undetected bugs, vulnerabilities, and regressions, potentially compromising both functionality and user experience.

The CSM can recommend two solutions here:

1. A thorough review of the Grade B and C repositories to identify recurring issues.
   
   
2. Implementing a more rigorous testing regimen and using automated testing tools to increase LoC coverage.

Codebase Issues

This is the problem a CSM would start with. Styling issues are the most prevalent issue in this codebase. Styling issues can lead to inconsistency in code readability, making collaboration more challenging and potentially increasing the time taken to identify and resolve more significant issues.

After that, we have error-prone issues. Issues here can lead to unexpected behaviors in the application, degraded user experience, system crashes, and even potential data loss or breaches, thereby affecting the overall trust and reliability of the software.

Then, we have 2.7% of the codebase susceptible to security issues. This will be a major red flag for the CSM that they want to highlight for engineering managers. This indicates potential vulnerabilities in the codebase that the team needs to investigate.

The CSM might suggest prioritizing a security audit for the repositories with the highest volume of security issues. They should consider implementing static code analysis tools explicitly focusing on security vulnerabilities and regular training sessions on secure coding practices for developers.

Top Repos

After these high-level issues, the CSM will take the team through repo-level problems. Some repositories have a high volume of issues, while others are at risk due to security issues.

The engineering manager should assign dedicated teams to the top repositories with the highest issues volume for a deep dive and remediation. They can also implement stricter code review processes for these repositories and consider using tools or plugins highlighting real-time security issues during code development.

PRs not up to Standard

Many PRs are not up to standard, with 913 closed and 51 still open. A CSM might suggest implementing a stricter PR review process and ensuring every PR undergoes a thorough review before merging.

Coverage Issues

85% of the code has no test coverage. A CSM would emphasize the importance of test-driven development (TDD) and regularly review test coverage reports. They might suggest allocating dedicated sprints to write tests for modules with no coverage.

With this report, a Codacy customer success manager would reinforce the idea that consistent monitoring and proactive measures can significantly enhance code quality. They will want to regularly review this dashboard, setting benchmarks and seeking continuous improvement to ensure a healthier, more secure, and maintainable codebase.

How Engineering Managers Can Use These Reports

Currently, the ROI Report dashboard is only available through CSMs (though if you think this is a feature you’d like access to, drop us a line).

But, on seeing this data, an engineering manager will immediately think about what they need to do to increase code quality. Their primary responsibility is ensuring the development team produces high-quality code, adhering to best practices, and delivering features on schedule. They need to know they can trust their engineers and codebase.

Looking at this dashboard, their primary concerns and subsequent steps would follow from the CSM's findings and recommendations, with a little more specificity:

1. Repo Quality: 
   • Concern: Many repositories fall under Grade B (64%), and the test coverage is notably low at 33.56%. This poses risks for maintainability and potential unseen issues in the codebase.
     
     
   • Next Steps: 
     • Organize a review session for Grade B and C repositories to understand the common quality pitfalls.
       
       
     • Advocate for and allocate resources toward improving test coverage. Encourage developers to write unit and integration tests in their development workflow.
       
       
2. Code Base Issues:
   • Concern: The high number of security issues (16,834) is alarming, given the potential risks associated with software vulnerabilities.
     
     
   • Next Steps:
     • Arrange for a dedicated sprint focusing on addressing critical and major security issues.
       
       
     • Explore and integrate additional security scanning tools to proactively identify and rectify vulnerabilities.
       
       
3. Top Repos:
   • Concern: Repositories with a high volume of issues might be central to our products or services, and their instability can pose operational risks.
     
     
   • Next Steps:
     • Assign focused teams to triage and resolve the issues in these top problematic repositories.
       
       
     • Initiate root cause analysis to understand recurring issues and prevent them in the future.
       
       
4. PRs not up to Standard:
   • Concern: Many PRs not meeting standards indicate potential gaps in the code review process.
     
     
   • Next Steps:
     • Implement mandatory code review guidelines.
       
       
     • Consider integrating automated code review tools that provide immediate feedback on PRs.
       
       
5. Coverage Issues:
   • Concern: With 85% of the code having no test coverage, there's a considerable risk of undetected bugs and issues.
     
     
   • Next Steps:
     • Advocate for a culture of test-driven development (TDD) within the team.
       
       
     • Plan dedicated sprints to increase test coverage, prioritizing critical components of the application.

Lastly, they should schedule a team meeting to discuss the dashboard's findings openly. It's essential to involve the engineering team in understanding the issues and collaborating on solutions, ensuring buy-in and collective ownership of the codebase's health.

Data Is at the Heart of Quality

We’ve designed this dashboard with one thought in mind—giving our customers access to the correct data to make the right decisions for the quality of their codebase. 

We want our CSMs to guide engineering teams to improve code quality, security, and coverage in their repos. By doing that, they can foster a culture of excellence, streamline development processes, and ultimately deliver products that stand out for their reliability, performance, and user satisfaction.

If you're new to Codacy, we encourage you to start a free, 14-day trial today to see how our platform can transform and optimize your code review processes.