5 Expert Tips on Secrets Management: Solutions, Tools, Dos and Don’ts

Every application works with secrets – sensitive data that can only be accessed by a limited number of users, applications, or processes. Common examples of secrets are user credentials and API keys, although a secret can be virtually any sensitive information that developers want to restrict access to. But managing secrets is challenging: there are just too many types of them and too many dedicated secrets management tools on the market. Which secrets management solution is best for your project?

In this article, we provide a brief overview of some of the most popular secrets management solutions: HashiCorp Vault and Consul, Microsoft Azure Key, Amazon AWS Secrets Manager, Kubernetes Secrets, and Docker Secrets. We also turn for help to our expert, Denys Bielyshev, to find out how to choose a secrets management solution that best fits the needs of your project.

Introduction to secrets

Secrets are basically any piece of data that you want to restrict and fully control access to. Common examples of secrets are passwords, API tokens, and configuration files.

Let’s take a closer look at the most common types of secrets.

Types of secrets

There are several ways we can classify secrets. For instance, based on the nature of the information they contain, we can divide them into three large categories:

• Sensitive security information (business data)
• Personally identifiable information (personal data)
• IT systems security information (IT data)

On the other hand, we can classify secrets by what they actually are:

• User credentials – Secrets, typically usernames and passwords, used for authenticating physical users and granting access to protected data, services, or endpoints.
• Database connection strings – Secrets used for granting an application access to a protected database. Such a string contains credentials with a clear text password and all the data needed for establishing a connection to the target database or file.
• Cryptographic keys – Secrets that contain encrypted data and can be used for different tasks, from data encryption to identity verification and user authentication.
• Cloud service access credentials – Secrets used for accessing services, applications, and servers of a particular cloud service provider. Cloud service access credentials can be either held directly by an application (if the application works with its own data) or obtained from an end user (if the application requires user information).
• Application programming interface (API) keys – Secrets meant for identifying the source of an API request. API keys can be either public or private.
• Access tokens – Client secrets with access tokens the OAuth or SAML client needs for authenticating with the Authorization Server or Identity Provider respectively. These access tokens define what data can be accessed by a specific client.

But why does secrets management matter so much?

Secrets are used by all kinds of applications for authenticating not only actual physical users but also specific processes and other applications. The more authentication processes are required, the more secrets are used. As a result, we face the need to securely and efficiently manage this highly sensitive data.

This is why secrets management is closely associated with Identity and Access Management (IAM). On the one hand, IAM solutions can help you verify and manage the identities of users, machines, and applications. On the other hand, IAM solutions allow you to set specific access permissions for both particular entities and groups of users.

Lack of security and proper access management, in turn, can lead to devastating consequences, including an increased risk of cyber attacks and theft of certificates or user credentials, such as by Man-in-the-Middle (MITM) attacks and spoofing.

In the next section, we talk about the main dos and don’ts of secrets management and determine the biggest challenges to this process.

Secrets management mistakes and challenges

Managing secrets entails numerous challenges. There are just too many factors that need to be taken into account, from determining the scope of secrets to be managed to actually choosing the most fitting secrets management solution or toolset.

Usually, the starting point for secrets management is determining where to store this extremely valuable data. At this stage, developers tend to choose the easiest and fastest ways of managing secrets. There are several obvious storage options:

1. Hard-code secrets into the source code
2. Enter secrets through the command line
3. Obtain secrets from a shared configuration file

Unfortunately, none of these options is secure and efficient enough.

Writing secrets directly to the source code has two major disadvantages:

• Secrets are stored in plain text, which is a great risk from a security point of view.
• The source code must be changed every time you need to update the secret written in it, which adds more complexity to the workflow.

The second option also can’t be called secure, since you’ll have to either enter secrets manually via the command line or use a script where secrets are stored in plain text.

Finally, when keeping secrets in a configuration file, there’s a significant risk associated with the way such a file is stored. It’s hard to properly limit access to files stored in a shared space, especially when they aren’t encrypted.

We summarize some of the most common mistakes when storing and managing secrets in the scheme below.

Aside from these bad practices, there are a number of challenges you need to keep in mind when thinking through the process of secrets management.

Key challenges to secrets management

The key challenges developers face when dealing with the need to store and effectively manage secrets are common not only for open-source and commercial solutions but for many self-developed tools as well. Let’s look at the challenges that are common for most secrets management tools and services:

• Secret lifecycle – The secret lifecycle consists of several stages: generation, rotation, assignment, sharing, and revocation. The main problem is that not every tool allows you to easily manage each of these stages.
• Secret revocation – Revocation is one of the most important stages in the secret lifecycle. If a previously released secret is compromised in any way, your tool must be able to revoke the associated security certificate the moment this compromise is detected.
• IAM integration – IAM is an essential part of secrets management. You should be able to easily integrate your solution with multiple authentication options. The most popular are OneLogin, Active Directory, Okta, OAuth 2.0, and AWS IAM.
• Secret storage approaches – A lack of innovative approaches to storing secrets creates additional security risks, as hackers are just as familiar with these techniques and mechanisms as developers. In most cases, keys and passwords are encrypted and stored in restricted storage or databases. Keeping secrets in distributed or decentralized storage could be an interesting alternative to the typical approach.
• Accessible APIs – Most solutions in this class are used for authenticating certain services and components of a target application. But for successful machine-to-machine authentication, it’s crucial for a secrets management solution to have accessible and comprehensive APIs.
• Reliability – A secrets management system should be reliable and have a high level of availability.
• Support for multiple infrastructures – As multi-cloud computing increases in popularity, developers face the problem of managing secrets across multiple infrastructures. The lack of support for external infrastructures in some native secrets management solutions becomes a real challenge.

Unfortunately, not all of these challenges can be addressed with the same secrets management solution. However, you can keep these pain points in mind when choosing the tool that best fits your needs.

In the next section, we compare some of the most common solutions for secrets management.

Top secrets management solutions: pros and cons

In this article, we focus on six popular and widely used secrets management solutions: HashiCorp Vault and Consul, Microsoft Azure Key, Amazon AWS Secrets Manager, Kubernetes Secrets, and Docker Secrets.

Let’s start our overview with the two solutions offered by the same provider: HashiCorp Vault and HashiCorp Consul.

HashiCorp Vault

Hashicorp Vault is a popular secret repository that supports a large number of authentication mechanisms and secret storage backends. Vault has useful features such as access control lists and time-to-live (TTL) secret access.

Complexity is the main drawback of Vault. There are too many elements and features to set up and maintain even in the basic version. However, the list of advantages is much greater. Let’s take a look:

• Access granularity – First and foremost, Vault ensures an impressively high level of data security. It separates different secret storage and authentication backends into completely independent units. You can define specific policies and TTL settings for each secret and ensure the required level of access granularity.
• Access security – Vault provides secure and comfortable access to any type of secrets, be they security certificates, passwords, database credentials, or API keys. At the same time, it records and saves a detailed activity log that can be used for security audits.
• Data encryption – Vault allows you to encrypt secrets before moving them to your chosen storage to ensure a higher level of data protection. Plus, you can use this tool for encrypting other data, even if it won’t be stored in Vault.
• Dynamic secret generation – Vault allows you to generate secrets for accessing some target points, such as SQL databases or AWS services, dynamically upon request. Dynamic generation of secrets is a secure alternative to providing users with unnecessary root privileges.
• Secrets renewal and revocation – Vault allows you to revoke secrets automatically (at the end of the lease associated with a particular secret) and manually. It also has built-in APIs for renewing secret leases.

For more details, you can turn to HashiCorp’s detailed Vault guide.

HashiCorp Consul

Technically, Consul is meant for service discovery, configuration, and monitoring. So there’s no point in comparing HashiCorp Consul vs Vault. However, as Consul supports access control lists (ACLs), it can also be used for restricting access to service information and storing keys.

The main security issue with Consul is the lack of encryption for data both in transit and at rest. Also, in contrast to Vault, Consul provides no built-in authentication or auditing mechanisms.

On the bright side, Consul has a high level of scalability and fault tolerance. So you can use it as a Vault backend to, say, store already encrypted information at rest.

AWS Secrets Manager

AWS Secrets Manager is a tool that AWS users can enable for storing and managing credentials and other sensitive information. Among other things, this tool can rotate AWS RDS credentials automatically on a schedule.

Compared to Vault, AWS Secrets Manager lacks a number of functionalities, including SSH CA authority and dynamic key generation. However, the platform provides two interesting alternatives for dealing with sensitive data:

• AWS Parameter Store is a free alternative to the standard Secrets Manager. This feature allows you to leverage HashiCorp Vault secrets management for your AWS application, only in this case AWS, not Vault, will manage everything. 
• AWS Key Management Service (KMS) isn’t a secrets management feature at its core. Rather, it’s a feature for efficiently encrypting information that needs to be protected. For instance, you can use AWS KMS to encrypt an encryption key. Then you can safely store the encrypted key with other secrets without any risk of exposing sensitive data.

Microsoft Azure Key Vault

Microsoft Azure provides a useful centralized service – Azure Key Vault – for managing any type of sensitive information used by Azure cloud applications. Key Vault has four major features:

• Secrets Manager – Secure storage and granular access control for all types of secrets
• Key Manager – Fast and easy creation and management of encryption keys
• Certificate Manager – Easy deployment and management of SSL/TLS certificates, both private and public
• Hardware Security Modules (HSM) – Secure storage of all secrets and keys can be ensured by either FIPS 140-2 Level 2 (which validates HSMs) or special software.

One of the main benefits of Azure Key Vault is that, while being a native secrets management solution for the Microsoft Azure platform, it can easily be integrated with other applications, even outside of Azure. Among its other advantages are:

• Granular authorization – Azure Key Vault uses role-based access control (RBAC) for granting access to vaults and access policies for granting access to the data stored in these vaults.
• No hard-coded secrets – The service provides centralized storage for all types of secrets. An application can’t directly access data stored in Azure Key Vault. The required data can be retrieved by an application with the help of Uniform Resource Identifiers (URIs).
• Detailed logs – Key Vault allows for monitoring and logging all actions related to secrets: who accessed them, when, and what was changed.

As for drawbacks, Azure Key Vault doesn’t allow the use of any special characters, such as underscores, in the names of keys and secrets, which may be inconvenient for some users.

Kubernetes Secrets

Kubernetes (K8s) also has a native solution for secrets management – Kubernetes Secrets. Basically, a Kubernetes Secret is an object that can carry any kind of sensitive information a web service might need to run smoothly and effectively. However, you can easily integrate Kubernetes Secrets with already running services and use native Kubernetes secrets as well as any other secrets. For instance, if you need to grant an application access to AWS or Azure services, you can either use native K8s secrets or launch Vault in a cluster and use it instead. 

Let’s take a look at some specifics of Kubernetes:

• Base64 encoding – One of the key features of Kubernetes keys is that they’re encoded in base64 format. Kubernetes provides built-in functionality for encoding information into base64 and decoding it back, so you can easily encode and decode sensitive data whenever you need to. The main problem is that everyone else can do it too. So to securely encrypt secrets and ensure they’re protected from unauthorized access, it’s better to use K8s native encryption feature (available for Kubernetes version 1.13 and above).
• Lack of visibility – When it comes to secrets, you need to be able to see who changed what and when. Kubernetes has an audit feature that’s disabled by default and provides a basic set of audit options. As an alternative, you can encrypt your secrets and move them to a private Git repository, which offers a much richer variety of auditing mechanisms and can provide you with better visibility into the history of secret changes. Just keep in mind that storing your sensitive data in a repository may create additional cybersecurity challenges.
• Data storage – Kubernetes secrets are stored in plain text and can be visible in transit. However, etcd supports SSL/TLS certificates. It’s better to configure etcd to use a TLS pair for traffic encryption.

Fortunately, Kubernetes can be effectively integrated with both third-party cloud solutions and open-source tools. You can easily pair Kubernetes Secrets with any of the solutions we’ve described above.

For instance, you can integrate Kubernetes with Vault and thus exclude the possibility of cloud vendor lock-in. Vault has a much richer functionality set than Kubernetes, and can work efficiently anywhere you can run Kubernetes. In particular, it supports a number of authentication methods, including multi-factor authentication (MFA), allowing you to increase the level of data protection when necessary.

Docker Secrets

Docker Swarm, a popular open-source platform for container orchestration, sees secrets as pieces of sensitive data that shouldn’t be:

• hard-coded into your application’s source code
• unencrypted at rest and in transit

The platform has a native secrets management solution called Docker Secrets. Similar to Kubernetes, in Docker Swarm, secrets are objects that carry sensitive information. Starting with version 1.13, Docker Secrets is available for Swarm clusters.

Here are some specifics of Docker Swarm secrets management:

• Access limitations – The Docker Secrets feature is only available in swarm mode. A standalone node can only have access to encrypted secrets, and only if it either is a swarm manager node or has a special access permission.
• Strong encryption – In Docker, secrets are encrypted both in transit (using a mutual TLS connection) and at rest (in an encrypted Raft log). A swarm manager saves a secret to a Raft log with a 256-bit key.
• High availability – When being changed, the Raft log containing secrets is automatically replicated across all other swarm managers, ensuring that all managers have access to the most recent versions of secrets.
• Secrets rotation – Docker Swarm Secrets allows rotating secrets without changing the source code of your application. All changes are made to the docker-compose file that contains the source and target of each secret. First, you add a new secret to Docker Swarm. Then you replace the old secret with the new one, as shown here.

However, Docker doesn’t provide a built-in solution for automatic secret rotation. Plus, if you want to make the versioning of your secrets trackable and manageable, you’d better think it through before creating the first secret. Adding version markers to secret names is one possible option to simplify the process of managing versions.

Also, note that Raft data isn’t encrypted in Docker version 1.12 and below. Make sure to update your manager nodes to Docker 1.13 or above; otherwise, your secrets will be stored in plain text.

But what option should you choose for your specific project? Is there a silver bullet for all cases, or does secrets management require a case-by-case approach? Let’s ask the expert.

5 expert tips on managing secrets

Trying to figure out which option works best for storing and managing secrets, we turned for help to our expert, Senior Developer Denys Bielyshev of Apriorit’s Driver Development Team. Together, we tried to unravel the truth about secure and efficient secrets management.

1) What is the best way to store secrets?

Denys Bielyshev: There’s no one-size-fits-all solution. Too many factors need to be taken into account. For example, security within a cluster has a low layer and a high layer. And the choice of method for storing secrets strongly depends on which layer of cluster security it will be used for. It’s also important to pay attention to the level of a secret’s availability.

Finally, you need to consider the provider of the machines used for deploying a particular environment. Many providers, such as AWS and Azure, have their own secrets management tools. And in most cases, using a native secrets manager is the best option in terms of convenience and security. However, this decision has one clear disadvantage – the high price of native secrets.

Consequently, HashiCorp Vault can be used in cases where you need to reduce the costs of storing and using secrets or when you don’t want to trust third parties with storing your sensitive data. Kubernetes can also be used in almost all cases, although in contrast to Vault, K8s secrets don’t get updated immediately. The time needed for Kubernetes secrets to be updated may vary from a few seconds to a couple of minutes, depending on the settings of a particular cluster.

As a result, to choose the best option, you’ll have to evaluate the performance, cost, and all possible pros and cons of several solutions. Then just choose the lesser of the evils. In some cases – for instance, in fully closed environments – the use of secrets may not be necessary at all. Although in general, the use of secrets is one of the main best practices for ensuring a proper level of data security.

2) What are the main requirements for secrets management solutions?

DB: While you should always choose a secrets management solution based on the particular needs of your current project, there are a number of basic requirements it should meet:

• Encryption at rest – While most secrets management solutions encrypt data in transit, not all of them ensure the same level of protection for data at rest. Give the nod to the solution that doesn’t store secrets in plain text under any circumstances.
• Encryption key export – The way a secrets management solution treats encryption keys is vital. If someone can get ahold of your encryption key, they can easily decrypt all of the data that was encrypted with it. So your best choice is a solution that either restricts encryption key export or accompanies it with additional security measures.
• Rotation of encryption keys – Encryption key rotation is among the key compliance requirements regarding the security of sensitive data. Make sure your solution can rotate encryption keys automatically on a schedule.
• Revocation of encryption keys – A compromised encryption key can lead to a devastating data breach, so it must be revoked as quickly as possible. The problem is that not all secrets management solutions have a built-in mechanism for encryption key revocation, leaving a significant gap in your data security system.
• Transparency of use – Since we’re talking about data with high priority, it’s best to be able to see when it was altered, how exactly, and by whom. Choose a solution that offers a wide range of auditing mechanisms.

3) What are the main security concerns when using secrets?

DB: One of the major security concerns regarding secrets is the appropriate configuration of IAM roles. For instance, by default, a secret should be provided to a specific application within your infrastructure. But if the roles are misconfigured, secrets can be accessed by an application that’s not supposed to have access to them.

Initially, when a secret is only created, it’s recorded using a role that has write access. But if someone with read access gets access to this file, they’ll eventually get access to data they aren’t supposed to have access to.

Therefore, it’s not enough just to choose a solution with fine IAM functionality. You need to spend some time and effort to properly configure all roles that provide different levels of access to secrets.

Another common problem is that developers don’t always pay enough attention to the way different application use secrets. For instance, even if your sensitive data is well hidden from everyone except for a specific role with the right access permissions, some application might still log secrets. These logs can eventually lead to the exposure of your sensitive information.

For example, if a microservice logs all incoming requests it receives, it may also log a valid authentication token. And since accessing such logs is much easier than getting access to the actual secrets, we face an additional cybersecurity risk here.

For instance, an attacker may use an application that automatically searches for authorization headers in logs stored by a microservice. Such an application can even run outside the target infrastructure and use exported logs to get access to sensitive information. If it finds an authorization token in these logs, this application can use it to send requests to services it initially didn’t have access to and, eventually, get ahold of restricted data. Even though tokens don’t live long, an attacking application will have enough time to pull at least some sensitive data stored in the compromised microservice.

Also, if a microservice isn’t designed properly, it may leave some sensitive data easily accessible by sending it via email alongside a notification, leaving it on a disk, sending it to another service, etc. Here are some basic recommendations for mitigating this risk:

• Keep the lifetime of secrets as short as possible.
• Make sure a secret can only be received by the entity that actually uses it.
• Once a service stops using or no longer needs a secret, revoke this service’s access to it immediately.

4) What are the biggest challenges of integrating secrets?

DB: Careful and correct configuration is the main difficulty in implementing any kind of secret. First of all, it’s important to appropriately define IAM roles. Then you need to properly distribute traffic to eliminate the possibility of exposed ports that can lead to data compromise.

Additional attention should be paid to determining access rights for various users, including the roles assigned to applications rather than physical users. On the one hand, it’s important for each particular role to have exactly as many rights as it needs – no more, no less. On the other hand, you need to ensure that different roles don’t duplicate or overlap each other in separate rights, which is extremely challenging.

While such planning is a complex and time-consuming task, this stage can’t be ignored – it’s just too important for ensuring a high level of secret security in the future.

One more common issue is attempting to introduce additional features or functionalities to standard secret managers. Of course, various alterations and refinements can probably help you solve a specific problem that a standard solution doesn’t cover. But on the other hand, it’s important to understand that when implementing such innovations, you should take all responsibility for any possible malfunctions of the infrastructure or data leakages.

Another problem, service availability, is more about supporting than implementing secrets – especially when we’re talking about on-premise solutions such as Vault and Consul. With AWS, Azure, and Kubernetes, availability is the concern of the cloud provider. But when using some software for managing secrets, it’s your responsibility, so you should create backups and think about ensuring smooth operation of your on-premise secret manager. Otherwise, there’s a risk that your entire infrastructure will collapse.

5) How can you improve or automate secrets management?

DB: The more secrets your application uses, the more operations you need to perform in order to manage them. Fortunately, there are high-level tools that can simplify secrets management. Such tools help automate the whole process and, consequently, reduce the amount of time you spend managing secrets manually.

There are a great variety of specifically designed tools for all popular secrets management platforms. For example, for AWS, there’s the boto3 Python framework. Using this framework, you can create a special script to automate separate secrets management processes. Similarly, there’s a command-line tool for Azure Key Vault, Azure CLI.

And for Kubernetes, there’s a special tool by Nirmata. It allows you to audit Vault secrets and effectively manage them across several containers.

Conclusion

Applications constantly work with numerous types of sensitive information: API keys, passwords, security certificates, and so on. All this data can be classified as secret and managed with the help of special tools called secrets managers.

There are a variety of platforms and solutions for you to choose from: the traditional Vault, popular cloud solutions provided by Amazon Azure and Microsoft AWS, and the highly integrative Kubernetes Secrets and Docker Secrets. However, there’s no silver bullet among these options, so you need to choose one with your current project’s security needs in mind.

At Apriorit, we’re passionate about data security and application development. Contact us to get started with your ambitious project!