Secrets Management: A Complete Guide 

IBM's Cost of a Data Breach 2023 report found that the most common cause of a data breach in 2022 was sensitive, private credentials that were either stolen or compromised in cyberattacks. The report also estimates the average cost of such attacks at $4.5 million. 

Sensitive credentials enabling access to your software are just one type of secret that development teams must protect. However, according to GitGuardian’s 2023 State of Secrets Sprawl Report, organizations are having difficulty doing so. The study found that the number of hardcoded secrets detected in GitHub commits increased by 67% in 2022 compared to 2021.   

As software infrastructure and methodologies continue to leverage automation to increase development speed and productivity, the number of secrets that need to be protected grows. 

Every non-human identity (scripts, automation tools, etc.) you add to your development process uses sensitive credentials to communicate with the rest of your environment, adding new credentials that can give bad actors access to secure systems—secrets that must be protected by creating and implementing various DevSecOps best practices, a process known as Secrets Management. 

What Are Secrets in Coding?

In everyday life, a secret is shared confidentially with a few entities who are obligated to protect what was shared from others gaining access to it. 

In software development, secrets refer to non-human sensitive credentials that provide digital authentication and grant access to software data and systems. As mentioned, software applications are no longer stand-alone systems. Applications rely on third-party databases, cloud infrastructure, software, and more—a vast number of components—all tied together and securely connected via secrets. 

Some of the most common types of secrets in software development include:

• User credentials like usernames and passwords used for user authentication and access control, as well as fingerprints and facial recognition data used for biometric authentication.
  
  
• Database connection strings necessary to establish connections between an application and a database, including credentials, server addresses, and database names.
  
  
• Cryptographic keys that encrypt and decrypt sensitive data, ensuring secure communication and user authentication.
  
  
• Cloud service access credentials required to access and interact with resources and services provided by cloud service providers.
  
  
• API Keys used as unique identifiers to authenticate and authorize applications accessing APIs (Application Programming Interfaces).
  
  
• Access tokens that provide temporary authorization for a specific user or application to access resources on behalf of that user.
  
  
• SSH keys used in the SSH (Secure Shell) protocol for secure communication and authentication between machines.
  
  
• Environmental variables, configurable values an application uses during runtime, which may include sensitive information like API keys or passwords.
  
  
• Configuration files containing settings and parameters used by an application which may include sensitive information and require protection.
  
  
• Session cookies used to maintain session information for user authentication and authorization.
  
  
• Security certificates used to establish the authenticity of a website, server, or user in secure communication protocols like HTTPS.

A secrets incident is a security event in which a secret is identified as a vulnerability and a potential threat to your organization. A secret incident usually has multiple occurrences, meaning the vulnerable secret exists across several files, repositories, or other locations within your infrastructure. 

The number of secret occurrences defines the potential impact of a secrets sprawl—the natural spreading of secrets throughout your application as the number of secrets distributed from developer to developer, application to application, etc., increases.  

Secret sprawl refers to the spread of secrets and the continuous decrease of your team’s ability to control and maintain visibility over secrets. When secrets sprawl across many components and systems, your software’s attack surface increases, meaning that the number of points where an unauthorized actor could potentially gain access to your data or systems grows. 

Modern applications rely on secrets to link all the moving parts necessary to run correctly. Every time you make a request to an API, link third-party applications to yours, or access a cloud database, you’re increasing the number of secrets you need to be aware of and protect adequately. 

So, how do organizations keep track of all these secrets and secure them effectively? Through the practice of secrets management. 

What Is Secrets Management?

Secrets management is a best practice for digital cybersecurity teams that gives your team better visibility into secrets and allows you to manage and enforce policies that keep all the non-human entities associated with your software in check. 

Secrets management aims to assure, across the entire tool stack, applications, and cloud databases, that secrets can only be accessed by those who are authorized and authenticated to do so. 

Some of the main initiatives associated with secrets management include: 

• Authenticating every access request that uses non-human credentials. 
  
  
• Making sure that the principle of least privilege (PoLP) and role-based access control (RBAC) are enforced; Concepts in data security that call for users to have the minimum level of access required to perform their jobs. 
  
  
• Tracking all access to secrets and maintaining a comprehensive audit of which person and application has access to what secrets. 
  
  
• When found, removing secrets from code, files, and other potentially unprotected attack surfaces. 

Think about secrets in software as keys to your home, car, or safe. Secrets management helps your organization ensure that keys that unlock your company’s most vital secrets don’t fall into the wrong hands. 

Why Secrets Management Is Important

When you manage secrets properly, you’re helping your organization avoid potentially costly, if not financially devastating, data breaches. As already mentioned, IBM’s most recent data breach report shows that the average cost of a data breach in 2023 was almost $4.5 million.

A good secrets management plan helps you create your own security model to improve your team’s capacity to store and audit all existing secrets. Secrets management should rely on automated systems and security tools to minimize the involvement and responsibility of humans and decrease the chances of human error (like successful social engineering attacks) resulting in secrets incidents. 

Your secrets management initiative should be defined and implemented as a systematic plan to avoid costly threats like data breaches and identity theft by preventing unauthorized access to your systems. 

As the complexity and diversity of your organization's systems continue to grow, the greater the chances of a secrets sprawl and the more critical secrets management becomes. As the number of secrets that need protection increases, secrets sprawls can be avoided by creating and implementing a united and all-encompassing secrets management system instead of relying on individual security models for managing secrets in each application, cloud database, etc. 

To maintain high visibility, secrets management policies should be managed centrally and applied consistently across your entire organization and every infrastructure component. Secrets management is also an ongoing process in which security leaders must continuously define rules and policies for how your organization deals with secrets at every stage of their lifecycles. 

What Does the Lifecycle of a Secret Look Like?

The lifecycle of a secret in software maps out the stages of its existence. Secrets management is all about handling the secret properly in every stage of its lifecycle, including:

• Creation. A user or system generates the secret, usually during the setup of a new application, user account, or any other scenario where authentication or authorization is required.
  
  
• Storage. Properly guarding the secret within your infrastructure through secure data storage mechanisms such as encrypted databases, key management systems, or secure configuration files. Examples of poor secrets storage would be hardcoding secrets in source code or storing them in plaintext.
  
  
• Distribution. Secrets are shared among the components or services needing authentication or authorization. Secrets management establishes secure mechanisms to transmit and distribute secrets internally, such as encrypted channels.
  
  
• Usage. Secrets are used during the regular operation of the software to authenticate and authorize access to resources, like when connecting to a database or calling an API. 
  
  
• Rotation. An essential aspect of secret management is establishing a system for regularly changing or rotating secrets. This helps minimize the risk associated with compromised or leaked secrets. Automated processes or tools can be used to manage secrets rotation and periodically update them.
  
  
• Monitoring. Continuous monitoring of secret usage and access is crucial for detecting unauthorized or suspicious activities. Logging mechanisms can help track when and how secrets are being used.
  
  
• Deletion. When a secret is no longer needed or has reached the end of its lifecycle, it should be securely deleted. This includes removing it from all storage locations and ensuring that backups or historical records do not contain the secret.

Secrets management also entails constant, secure, and up-to-date documentation of secrets' existence, purpose, usage, and eventual retirement for future reference. 

Challenges of Secret Management 

The greatest challenge of secret management is one you can’t control; the undeniable fact that cyberattacks on organizations of all sizes are increasing yearly in frequency and sophistication. 

Cybersecurity Ventures expects cybercrime to cost companies an estimated $10.5 trillion annually by 2025, with a year-over-year growth rate of 15%. 

Sensitive data is currency to cybercriminals. It can be used to access financial accounts and transfer money out of them. Uncovered secrets can also be used to extort money from companies, with bad actors taking hostage of sensitive data and requesting that companies pay a ransom for its safe return. 

Cybercriminals have an increasingly diverse and powerful tool stack (including machine learning and artificial intelligence) at their disposal. Cyberattack methods and forms are constantly changing and innovating, making it increasingly challenging for companies to identify and prevent them.

According to recent research by 1Password, IT/DevOps employees whose companies lost secrets also lost an average of $1.2 million, with 10% of those surveyed saying that their companies lost more than $5 million. 

The report also states that 40% of companies that experienced secret leakage experienced significant reputational damage, and almost 30% said they lost customers. 

However, there are plenty of internal challenges related to secrets management that your organization can control. Most of these challenges are linked to the rapid pace at which software development is evolving, and software infrastructure is expanding.

Secrets Are Everywhere 

With so many components working together in modern applications, secrets are widespread. They are found in internal and commercial solutions, servers, cloud databases, containerized applications like Red Hat OpenShift or Kubernetes, automated processes like Puppet or Chef, continuous integration/continuous deployment (CI/CD) toolchains, and more. 

Secrets are even found in the security software and vulnerability scanners you use to keep your secrets safe. 

According to 1Password’s survey, 65% of DevOps workers said their companies have more than 500 secrets, and almost 20% claim they have more than they can count. 

While certainly not as expensive as a data breach, the process of secrets management can also be costly. 1Password found that DevOps and security personnel spend 25 minutes a day on average managing secrets. That’s an estimated payroll expense of $8.5 billion annually in the U.S. alone. Secrets leakage can also lead to lost productivity, with DevOps telling 1Password that poor secrets management causes delays in over 60% of their ongoing projects. 

Decentralization

Unfortunately, secrets management isn’t always taken as seriously as it should be. 1Password’s report found that 80% of the DevOps employees they polled admitted to not managing their secrets well. 

One of the most common practices that makes secrets management more difficult is organizations decentralizing the process instead of uniting it under one roof. When developers, admins, DevOps, and other team members manage their secrets separately, overall visibility suffers. 

This siloed secrets management effort doesn’t provide holistic visibility across all layers of your infrastructure. As a result, auditing secrets becomes more difficult, and instances of security gaps and vulnerabilities increase. 

Secrets Are Often Hard-Coded 

It’s somewhat shocking to see how often applications contain hard-coded secrets and sensitive credentials, which hackers can easily uncover using scanning tools and then crack through basic brute-force or dictionary attacks. Hardcoded secrets can be easy to find and extract for cybercriminals because they are often stored in plain text. 

GitGuardian scanned over one billion GitHub commits for their 2022 report, revealing that 1.35 million users accidentally exposed a secret in their commits. The study found that almost six out of every 1,000 commits exposed at least one secret, a 50% increase compared to 2021. 

In their report, GitGuardian categorized two types of embedded secrets: specific (google_api_key, private_key_rsa, private_key_generic, googlecloud_keys, etc.), which accounted for 33% of the secrets that were found hardcoded, and generic (company emails, usernames, passwords, etc.), which accounted for 67%. 

Growing Tool Stacks and Reliance on Automation 

The pure number of secrets being used across increasingly numerous tools and systems amplifies secret management issues, especially in the DevOps environment, in which teams use dozens of tools and technologies for configuration management and orchestration that primarily run on automated scripts. DevOps teams must also account for third-party vendor accounts and remote access solutions in their secrets management process. 

According to 1Password’s report, 51% of DevOps said that the time they spend on managing secrets increased in 2022, with 10% saying that the time they dedicate to this process has more than doubled. More than half say that the increased proliferation of cloud applications has made managing secrets more difficult for their team, and 25% say that their companies have secrets in 10 or more locations. 

Secrets Management Best Practices 

While it’s important to follow best practices for secrets management, breaking bad habits is even more critical. As 1Password’s report shows, plenty of IT/DevOps employees are still doing a lot of things wrong when managing software secrets:

• 64% admit to reusing enterprise secrets between projects
  
  
• 36% share secrets over insecure channels to increase productivity and speed
  
  
• 59% have shared secrets with coworkers via email 
  
  
• 40% have shared secrets with coworkers via chat applications
  
  
• 36% have shared secrets with coworkers via spreadsheets/shared documents
  
  
• 26% have shared secrets with coworkers via text/SMS

And it’s not just an employee problem; organizational leaders and management also contribute to poor secret management. 1Password’s report found that while almost every organization has a secrets generation and management policy, only 36% say their companies strictly enforce these policies. Half of surveyed employees also said they have “explicit fears with how their company currently handles secrets.”

The report found that 63% of team leaders and 67% with VP and above status have ignored security policies to meet work demands. It also found that 81% of DevOps leaders and 65% of team leads/managers have reused secrets between projects.

But enough of what organizations are doing wrong. Here are some best practices to follow if you want to implement a successful, effective, and holistic secret management policy:

1. Implement RBAC to enforce the principle of least privilege and strict access controls by assigning access rights based on job responsibilities. Regularly review and update access permissions to ensure only authorized individuals or systems can access sensitive credentials.
   
   
2. Regularly rotate secrets to minimize the impact of a potential breach. Automate the rotation process where possible to ensure consistency and decrease the risk of human error.
   
   
3. Encrypt data in transit and at rest so that sensitive data and secrets are encrypted during transmission between systems and when stored in databases or other repositories. Use robust encryption keys and algorithms and regularly update encryption protocols to align with industry standards. 
   
   
4. Monitor and alert on anomalies to notify security teams of suspicious activities, such as multiple failed login attempts or access from unfamiliar locations.
   
   
5. Embrace automated secrets detection tools designed to scan code repositories, configuration files, and other relevant sources for potential leaks of sensitive information. Regularly run these tools as part of your CI/CD pipeline to catch and remediate secrets before they make their way into production.
   
   
6. Create backup and recovery procedures to ensure data availability and integrity in the event of system failures. Test the backup and recovery procedures regularly to verify their effectiveness.
   
   
7. Create a centralized vault. Use your automated tools to identify all private encryption keys, private certificates, and passwords in your system and add them to a centralized location, a secrets vault, and be sure to continue discovering and adding new credentials to the vault as they are created. 
   
   
8. Regularly audit and review policies of your secret management system to identify deviations from established policies. Review and update security practices and policies based on changes in the threat landscape, business requirements, or compliance standards.
   
   
9. Educate and train teams by providing comprehensive training on secrets management best practices for development, operations, and security teams. Promote a culture of security responsibility and awareness across the organization.
   
   
10. Vet third parties to ensure your vendors and partners follow best practices are aware of your secrets management policies and standards. 

Prioritize Early Secrets Detection to Optimize Security 

Secrets management policies serve as crucial foundationsfor a strong DevSecOps strategy. When your most sensitive data is safe and secure, your team is already on a path toward achieving a robust security posture that spans the entire development lifecycle.

With advanced security features, including secret and insecure dependency detection, Codacy empowers developers to catch and address potential security vulnerabilities early in development.

By seamlessly integrating security into your code review practices, Codacy becomes an invaluable ally in ensuring that your applications are functional and resilient against the ever-evolving landscape of cyber threats. Start a 14-day trial today to see how Codacy can automate and simplify your code quality and security processes.