Key Application Security Metrics You Should Be Tracking in 2024

Companies are increasingly prioritizing security to combat the growing threat of cyber attacks. Our 2024 State of Software Quality report shows that over 88% of companies have dedicated security teams, and over 84% conduct regular security audits.

Application security (AppSec) protects applications from threats by addressing vulnerabilities throughout the software development lifecycle (SDLC). An effective AppSec program uses technology and processes to identify, prevent, and resolve security vulnerabilities.

But how do you measure the effectiveness of your AppSec program? The answer lies in AppSec metrics. This article will discuss five essential AppSec metrics, explain why they matter, and how to start tracking them. 

Why is Tracking AppSec Security Metrics Important?

Tracking AppSec security metrics is essential for organizations to make data-driven decisions and ensure their security efforts align with overall goals. 

Tracking AppSec metrics helps answer questions such as:

1. Improvement Over Time: Are we making progress in our AppSec program? Tracking metrics allows you to see trends and determine if your security measures are becoming more effective.
   
   
2. Return on Investment: Are our investments in tools, personnel, and processes yielding results? By measuring the effectiveness of your AppSec program, you can assess whether your investment in security programs is productive.
   
   
3. Risk Identification and Management: Are we identifying and managing risks effectively? Metrics provide insights into areas where risks are being mitigated and where they may still exist.
   
   
4. Compliance and Standards: Are we meeting industry standards and compliance requirements? Tracking metrics helps ensure your security practices align with regulatory and industry benchmarks.
   
   
5. Resource Allocation: Are we allocating our resources efficiently? By understanding the security landscape of your applications, you can better direct your resources to where they are needed most.

5 Essential AppSec Metrics to Track For Security

Understanding and tracking AppSec metrics is crucial for making informed decisions to enhance your security program. While no single metric can provide a complete picture, tracking a combination of key metrics can offer valuable insights into your security posture and help identify areas for improvement. Here are some essential ones.

1. Code Coverage

Code coverage measures how much of the codebase is executed during automated testing. It is expressed as a percentage calculated by comparing the number of code lines covered by tests to the total number of lines in the codebase.

Low code coverage indicates sections of your codebase that haven't been exercised by tests. These untested areas are prime breeding grounds for vulnerabilities, as hidden flaws might remain undetected. Thoroughly tested code with high coverage is generally better understood and more maintainable. This translates to a lower risk of introducing new vulnerabilities during future modifications. Developers can refactor and modify code with greater confidence, knowing the existing functionality remains intact.

Many languages have code-coverage libraries that easily integrate with testing frameworks to help generate code-coverage reports for your database.

• JaCoCo (Java Code Coverage) is a free code-coverage library for Java that integrates well with Java unit test frameworks like JUnit.
  
  
• coverage.py is a built-in Python module that integrates with many testing frameworks, such as pytest. It offers detailed reports and integrates seamlessly with Python development workflows.
  
  
• NUnit is a popular testing framework for C# with built-in code coverage features.

You can link your code coverage reports to Codacy if you have a large codebase. This lets you view code coverage reports from different languages and projects in one place, enabling easy comparison and trend analysis.

Integrating this with your CI/CD pipelines allows for automated code coverage checks on each commit.

2. Number of Vulnerabilities and Their Severity

Tracking the number of vulnerabilities in your applications and their severity levels provides insights into the effectiveness of your security practices and helps identify trends in your application security over time.

Tools like Codacy offer static application security testing (SAST) and dynamic application security testing (DAST) functionality to pinpoint parts of the application susceptible to vulnerabilities such as SQL injection, cross-site scripting (XSS), and other OWASP Top 10 issues.

After running a scan, classify vulnerabilities based on their severity level: critical, high, medium, or low. The severity level reflects the potential impact a vulnerability can have if exploited. Critical and high-severity vulnerabilities should be addressed as top priorities, while medium and low-severity vulnerabilities can be managed based on risk tolerance and resource constraints.

Tracking trends and the number of vulnerabilities discovered at each severity level helps you gauge the progress of your security efforts. For example, if SQL injection vulnerabilities appear less frequently, it suggests that your developers are writing more secure code, which is a positive sign for your team.

The Codacy AppSec dashboard makes it easy for your team to prioritize which security issues to focus on first, showing which repos are most at risk and which types of security issues are most prevalent. We've also recently added a severity filter. 

3. Mean Time to Discover Vulnerabilities 

Measuring the mean time to discover vulnerabilities (MTTD) is crucial in evaluating how quickly your security team can identify potential issues. A shorter MTTD indicates a more proactive security posture and a reduced window of exposure to potential threats.

To calculate MTTD, you need to determine the average time it takes to detect a defect after it’s introduced. To do this, you need to:

1. Identify each security incident and record the following timestamps:

• • When the vulnerability was introduced (from the code commit timestamp)
    
    
  • When the vulnerability was discovered (when your security team noticed it or an automated detection tool flagged it)

2. Calculate the Time to Detect (TTD) for each incident by subtracting the introduction timestamp from the discovery timestamp

3. Find the overall MTTD by adding up the individual TTDs for a specific period (e.g., month, quarter) and dividing by the total number of incidents

MTTD = ∑(Time Detected−Time Introduced) / Total Number of Vulnerabilities

Let's say your team detected three vulnerabilities in the past month. The TTDs were:

• Vulnerability 1: 5 days
• Vulnerability 2: 3 days
• Vulnerability 3: 2 days

Using the formula:

MTTD = (5+3+2) / 3 ​≈ 3.33 days

In this example, the MTTD is approximately 3.33 days.

If vulnerabilities are identified before being pushed to production (e.g., through a SAST tool running in your CI/CD pipeline), the Time to Detect (TTD) should be considered zero (0). This metric can incentivize developers to adopt assessment tools earlier in the development lifecycle, leading to earlier detection and mitigation of potential security issues.

4. Mean Time to Remediate Vulnerabilities

Mean Time to Remediate (MTTR) is arguably the most critical metric in application security. Finding vulnerabilities is only the first step; fixing them promptly is what matters. MTTR measures the average time it takes to fix a vulnerability once identified.

Like MTTD, follow these steps to calculate the MTTR: 

1. Identify each security incident and record the following timestamps:

• When the vulnerability was discovered (when your security team noticed it or an automated detection tool flagged it)
• When the vulnerability was fixed

1. Calculate each incident's Time to Remediate (TTR) by subtracting the discovery timestamp from the remediation timestamp
   
   
2. Find the overall MTTR by adding the individual TTRs for a specific period and dividing by the total number of incidents

MTTD = ∑(Time Fixed−Time Discovered) / Total Number of Vulnerabilities

A lower MTTR indicates a faster, more agile response from your team. You can use project management tools like Jira and Trello to create tickets for each identified vulnerability, log the discovery time, and track the time taken until the ticket is resolved. Integrating these tools with your dashboards and asset inventory systems provides a comprehensive view of your remediation efforts, helping you to streamline processes and reduce MTTR.

5. Employee Security Training and Awareness Metrics

Humans often represent the weakest link in security, making employee training and awareness a vital component of your security strategy. Regular training helps employees recognize and respond to security threats, reducing the risk of social engineering attacks.

Track participation rates, quiz scores, and incident response performance to measure the effectiveness of your training programs. Use platforms like KnowBe4 to conduct phishing simulations and other training exercises, then analyze the results to identify areas for improvement.

Maintain a schedule of regular training sessions, ensuring all employees, from developers to executive leadership, understand their role in maintaining security. Keep training content up-to-date with the latest threats and best practices.

Some Factors to Consider When Setting Up AppSec Metrics Tracking

When establishing an AppSec metrics tracking system, several critical factors can enhance the effectiveness and relevance of your efforts:

1. Understand Your Environment: Without a comprehensive understanding of your assets and their interconnections, managing security becomes guesswork. Knowing your applications and services and how they interact is crucial for identifying potential vulnerabilities and attack surfaces. This ensures that all components are accounted for and monitored.
   
   
2. Track Over Time: Avoid static representations like pie charts. Instead, focus on trends over time. Tracking metrics over a period of time provides insights into patterns and progress, making it easier to identify areas for improvement and assess the impact of your security measures.
   
   
3. Contextualize Results: Present metrics in a context that highlights their importance and impact on the organization’s bottom line. For example, quantify the cost of security incidents by tracking metrics like the number of customers lost due to a data breach, the associated hours and employee dollars spent, and the frequency and impact of ransomware events.
   
   
4. Establish Baseline Policies: Before tracking security metrics, it's crucial to establish baseline policies. These policies provide a reference point for your measurements, ensuring consistency and clarity in what you're assessing. Frameworks like the NIST Cybersecurity Framework offer comprehensive guidelines for setting these policies, helping you measure essential items effectively. 
   
   
5. Leverage Tools and Technology: Utilize advanced tools and technologies to automate and enhance your metrics tracking. Automated tools can provide real-time data, reduce manual effort, and increase the accuracy and reliability of your metrics.
   
   
6. Set Realistic Targets: Setting targets for your security metrics is crucial for engaging your team and driving continuous improvement. Ensure that these targets are realistic and achievable, as unrealistic goals can demotivate your team and lead to inaccurate reporting. For instance, if you aim to reduce the number of critical vulnerabilities by 20% over the next quarter, ensure this target is based on a thorough assessment of your current capabilities and resources, not just some random figure. Realistic targets encourage your team to strive for excellence while maintaining morale and accuracy in reporting.

Enhance AppSec Metrics Tracking With Codacy

Tracking and analyzing AppSec metrics provide crucial insights into your organization's security posture. These metrics can help predict and prevent vulnerabilities and data breaches when properly monitored. Codacy is a comprehensive platform that can significantly enhance your AppSec metrics tracking efforts.

Codacy offers SAST and DAST capabilities, enabling you to detect and measure security vulnerabilities in your code. Codacy also integrates with popular unit-test frameworks to provide detailed code coverage reports so you can monitor the extent of your test coverage.

Codacy dashboards offer a clear and concise overview of your code quality and security metrics, allowing you to track trends over time, identify areas that need improvement, and measure the effectiveness of your security initiatives.

Get started with a free trial today.