Microservices and Container Security: 11 Best Practices

Today’s software is a mosaic: different services and features are responsible for delivering different capabilities. To make this complex software flexible and resilient, development companies often use microservices and containers. In a 2021 survey by Statista, 34% of respondents said they had adopted microservices, and 37% stated they used microservices partially, proving the popularity of this development approach.

Although a microservices approach has numerous benefits including easy scaling and management, it also hides some security concerns, such as possible vulnerabilities and communication risks. Knowing what cybersecurity risks to expect and how to address them before you start the development process will help you create a reliable and secure product.

In this article, we observe the security challenges in container and microservices-based architectures that might arise when developing applications. We also offer 11 microservices and container security best practices. This article will be useful for those who want to efficiently improve the security of their products and those who are new to containers and microservices.

What are containers and microservices?

Containers and microservices are popular approaches to application development, especially for complex solutions. O’Reilly’s Microservices Adoption in 2020 Report shows that 77% of respondents had adopted microservices, with 92% experiencing success with this approach. Moreover, respondents who used containers to deploy microservices were more likely to report success than those who didn’t.

Before we discuss how to implement security in microservices and containers, let’s explore exactly what these approaches are and what benefits they bring to the application development process.

Containerization is a form of virtualization that enables you to run applications in isolated spaces called containers that use the same shared operating system (OS). While virtualization allows for running multiple OSs on the hardware of a single physical server, containerization allows you to deploy multiple applications using the same OS on a single virtual machine or server.

Containers, also known as application containers or server application containers, are executable units of software that contain application code along with its libraries and dependencies. According to the National Institute of Standards and Technology (NIST), the fact that application containers are isolated from each other but still share the resources of the underlying OS makes it easier for developers to efficiently scale applications across clouds.

Developers prefer to go with a container-based architecture because containers are lightweight, portable, and easy to maintain and scale. Thanks to these qualities, containers can be used for modern development approaches like DevOps, serverless, and microservices. Also, developers can granularly control how many resources each container can use, improving CPU and memory utilization of physical machines.

A microservices architecture, or simply microservices, is an architectural approach to application development in which a single application is composed of many small, autonomous services.

Microservices are independently deployable, which allows you to improve application code, add new features, and scale each service much more easily than in a monolithic architecture. With microservices, you can update an existing service without rebuilding and redeploying the entire application.

Each service represents a separate codebase, so it can be managed by a small development team. A microservices architecture simplifies the process of creating and maintaining complex applications, but is not suitable for small applications.

Microservices are loosely coupled, so if one service fails, the rest keep working, improving the fault tolerance of the entire application. Moreover, they support polyglot programming, which means that services don’t need to share the same technology stack, libraries, or frameworks.

You can use containers and microservices separately or together. Since in this article we discuss security practices for developing applications that use both approaches, let’s briefly explore how containers and microservices can be combined.

In simple terms, a container encapsulates a lightweight runtime environment for an application. So when a microservice is developed within a container, it inherits containerization benefits like portability, scalability, and additional security layers. Containers provide isolation for each containerized application or microservice. Thus, they reduce the risk of security vulnerabilities spreading.

By running microservices in separate containers, you can deploy them independently, regardless of the language in which each microservice is written. This way, containerization removes the risk of any friction or conflict between languages, libraries, and frameworks.

In terms of service discovery, containerization makes it simpler for microservices to locate and communicate with each other, as they all run in containers located on the same platform. For the same reason, it’s also easier for developers to orchestrate microservices.

Despite all the advantages, both containers and microservices have their nuances and challenges, including cybersecurity matters. Let’s discuss some of the most important security concerns.

Security challenges of applications based on containers and microservices

As with any other approach, both containers and microservices can bring certain security challenges to application development. And to ensure decent cybersecurity of your product, it’s essential to be aware of the most common security risks and plan how to prevent and mitigate them.

1. Vulnerabilities that can be exploited

In general, discussions of security concerns related to microservices and containers have shifted to security requirements of orchestration platforms. However, not all security risks can be dealt with at the orchestration level. For instance, it’s essential to keep an eye on possible vulnerabilities that can be exploited.

1. Image vulnerabilities are the most common security threats within applications based on microservices and containers. They typically arise from insecure libraries or other dependencies. Vulnerable images themselves don’t pose an active threat. However, when a container is based on a vulnerable image, it will introduce the vulnerability to the entire environment.

2. Application vulnerabilities might appear from flaws inside the app’s source code. For example, if one of your applications has a buffer overflow vulnerability, attackers might exploit it to execute malicious code and take over your container.

3. Vulnerabilities to cyber attacks. Microservices-based applications are more complex than monolithic applications, as they consist of many moving parts. One application can include hundreds of microservices that are deployed in thousands of containers. For developers, this means that monolithic code with 1,000 dynamic-link libraries (DLLs) should be decomposed into the same number of microservices. It makes the microservices-based application quite vulnerable to cyber attacks because it’s hard to ensure proper security of that many components.

2. Malware risks

Possible scenarios for malware attacks include:

1. Hackers gain access to a container and inject malicious code into it that can also attack a microservice within this container, other containers, or the host operating system.
2. A malicious actor compromises your CI/CD environment and injects malware into the source code repositories that are used to build container images.
3. Attackers breach the container registry and replace images with ones that contain malware.
4. Hackers trick developers into downloading malicious container images from external sources.

The problem with malware is that if you don’t detect it before launching a container, the malware will infect your microservices within this container and the entire environment as well. For example, it can collect sensitive data, block processes, or disrupt the work of other containers.

3. Risks related to access to code

The more people can access and alter code, the more security risks arise. The two most common are:

1. Too broad of access rights. Many development companies choose the DevOps approach for building applications using microservices and containers because it breaks down the barriers between teams and ensures continuous integration and continuous deployment (CI/CD). However, DevOps can cause too broad of access rights, increasing the risk of someone altering code in the distributed working environment.

2. Weak secrets management. Even more people can get access to containers in case of poor security practices or security rules violations. For instance, developers might place hard-coded credentials in scripts into containers, or they might store secrets in an insecurely configured key management system.

4. Unrestricted communication between containers

Usually, containers can’t access any resources outside of the environment they directly control — this is called unprivileged mode. Engineers should only allow communication capabilities between containers that are necessary for correct application work. For example, an application container might make a connection to a database container.

Whenever a container has more privileges than is strictly required, it can cause additional security risks. A container can get excessive privileges as the result of misconfigurations caused by lack of experience or orchestrator mismanagement.

5. Managing data securely

The distributed framework of a microservices architecture makes it more challenging to secure data because it’s difficult to control access and secure authorization to individual services. Thus, engineers have to pay more attention to how their application ensures data confidentiality, privacy, and integrity within each service.

Another issue is that data in microservices is continually moved, changed, and used in different services for different purposes, which creates more entry points to data for malicious actors.

6. Choosing and configuring tools

When developing and maintaining a microservices architecture, DevOps teams use lots of tools, including open-source and third-party. While such tools help engineers achieve the efficiency needed for DevOps pipelines, they don’t always provide the required security.

If you don’t carefully assess the security capabilities of open-source tools you plan to integrate into your environment, you risk creating vulnerabilities within microservices and containers. And even if a tool seems secure enough, you still should be careful when configuring its settings and keep evaluating the tool’s security with time.

It can be challenging to choose appropriate security measures to address the risks we’ve mentioned above. The reason is that both containers and microservices are so attractive to developers because of the way they simplify and accelerate application development. Security measures are likely to be ignored or neglected if they make the development approach slower and less straightforward.

To help you leverage containers and microservices for application development without sacrificing cybersecurity, we’ve gathered 11 helpful practices. Let’s explore how to secure microservices and containers in detail.

11 best practices for securing microservices and containers

The practices we mention below can be a helpful addition to secure your current way of developing applications using containers and microservices. However, if you’re only starting to create an application or are about to migrate your product from a monolithic architecture to a microservices-based architecture, make sure to have comprehensive security strategies prepared.

NIST recommends outlining two types of strategies:

1. Security strategies for implementing core features	2. Security strategies for countering microservices-specific threats
Identity and access management Service discovery Secure communication protocols Security monitoring Resiliency or availability improvement techniques Integrity assurance improvement techniques	Threats to service discovery mechanism Internet-based attacks Cascading failures

You can find a detailed description and technical details of what to include in these strategies in the NIST SP 800-204 Security Strategies for Microservices-based Application Systems.

Our list of containers and microservices best practices is based on expertise of the Apriorit team, security recommendations provided by leading microservices practitioners, and industry standards as reflected in the following documents:

• NIST Special Publication 800-180 (Draft) NIST Definition of Microservices, Application Containers and System Virtual Machines
• NIST Special Publication 800-190 Application Container Security Guide
• NIST Internal Report 8176 Security Assurance Requirements for Linux Application Container Deployments
• NIST Special Publication 800-204 Security Strategies for Microservices-based Application Systems
• Department for Work & Pensions Security Standard – Microservices Architecture (SS-028) [PDF]

Now let’s look closer at how you can develop secure applications when using microservices and containers.

1. Create immutable containers

Developers tend to leave shell access to images so they can fix them in production. However, attackers often exploit this access to inject malicious code. To avoid this, create immutable containers.

According to the Google Cloud Architecture Center, an immutable container can’t be modified. If you need to update the application code, apply a patch, or alter configurations, you can rebuild the image and redeploy the container. If you need to roll changes back, you only need to redeploy the old image. You can deploy the same container image in every one of your environments, making them identical.

Note that remote management is done through runtime APIs or by creating remote shell sessions to the host on which the microservices are running. And the immutable nature of containers also affects data persistence. Developers should store data outside containers so that when some are replaced, all data is still available to their new versions.

2. Deploy one microservice per host

According to microservices.io, there are six patterns for deploying microservices:

• Multiple service instances per host
• Service instance per host
• Service instance per VM
• Service instance per container
• Serverless deployment
• Service deployment platform

The two most beneficial deployment options are service instance per container and service instance per host, since they allow you to:

• Isolate service instances from one another
• Eliminate the possibility of conflicting resource requirements or dependency versions
• Allow a service instance to consume at most the resources of a single host
• Easily monitor, manage, and redeploy each service instance

While deploying several microservices on the same host allows for more efficient resource utilization than the service instance per host pattern, it also has a lot of drawbacks:

• Risks of conflicting resource requirements and conflicting dependency versions
• Difficulties with limiting resources consumed by a service instance
• Difficulties with monitoring the resource consumption of each service instance if multiple service instances are deployed in the same process

3. Integrate automated security testing into your build or CI/CD process

There are various tools that can automatically test containers during the build or CI/CD processes. For instance, HP Fortify and IBM AppScan offer dynamic and static application security testing.

You can also use scanners like JFrog Xray and Black Duck to check containers for known vulnerabilities in real time. Once these tools discover vulnerabilities, they flag builds with detected issues, allowing you to check and fix them.

4. Avoid using privileged containers

If a container runs in privileged mode, it has access to all components on the host. Thus, such a container acts as part of the host operating system and impacts all other containers running on it. If such a container gets compromised, an attacker would have full access to the server.

Therefore, consider avoiding the use of privileged containers. For example, in Kubernetes, you can forbid privileged containers using Policy Controller.

If for some reason you need to use privileged containers, Google Cloud Architecture Center offers some alternatives:

• Providing specific capabilities to the container through the securityContext option of Kubernetes or the –cap-addflag option in Docker
• Modifying application settings either in a sidecar container or in an init container
• Modifying the sysctls interface in Kubernetes using the dedicated annotation

5. Run images only from trusted sources

There are many open-source packages for developers with readily available containers. However, for security purposes, you need to know where containers originate, when they were updated, and if they’re free of any known vulnerabilities and malicious code. It’s best to establish a trusted image repository and run images only from that trusted source.

In addition, developers should check application signatures in their scripts before putting containers into production. If you run containers across multiple cloud environments, it’s acceptable to have several image repositories. If you want to use images from other sources, it’s recommended to scan the images with scanning tools.

6. Use registries to securely manage images

Registries like Docker Hub, Amazon EC2 Container Registry, and Quay Container Registry help developers store and manage images they’ve created. You can use these registries to:

• Provide role-based access controls
• Specify trusted sources of containers
• Create and update the list of known vulnerabilities
• Flag vulnerable images

Note that role-based access controls are also important, as you need to control who can introduce changes to your containers. It’s best to limit access to specific administrative accounts: one with responsibility for system administration and one for operating and orchestrating containers.

Another thing to keep in mind is ensuring that your registry verifies the signature of each container and accepts only those that come from trusted sources. Also, leverage features that help you constantly check image content for known vulnerabilities and inform about security issues. For example, Docker Hub Vulnerability Scanning automatically scans the image to identify vulnerabilities in your container images once you push an image to Docker Hub and enable vulnerability scanning.

7. Harden the host operating system

While most recommendations refer to the security of microservices and containers, it’s also necessary to ensure the security of the host operating system.

First of all, NIST recommends using container-specific host operating systems (minimalist host OSs explicitly designed to only run containers), as they’re free of unnecessary functionality and thus have a much smaller attack surface than general-purpose hosts. It’s also preferable to use a platform that allows for controlling egress traffic with a router or firewall.

Secondly, CIS Docker Benchmark offers checklists for hardening your system. The recommendations vary depending on the operating system, server software, cloud providers, mobile devices, network devices, and desktop software.

The key recommendations are to:

• Establish user authentication
• Set access roles
• Specify permissions for binary file access
• Collect detailed audit logs

In order to avoid data compromise, limit container access to underlying operating system resources and isolate containers from each other. A good practice is to run the container engine in kernel mode while running containers in user mode.

For example, Linux provides technologies like Linux namespaces, seccomp, cgroups, and SELinux for securely building and running containers.

8. Protect microservices with the defense in depth approach

Defense in depth is an information security approach that relies on a combination of security mechanisms and controls like antivirus software, firewalls, and patch management. Its goal is to provide multi-layered security throughout IT systems to protect the confidentiality, integrity, and availability of networks and data.

The three key layers of the defense in depth approach are:

• Physical controls — anything that physically limits access to IT systems such as guards and CCTV systems
• Technical controls — hardware and software that aim to protect systems and resources (discussed below)
• Administrative controls — various policies and procedures to ensure proper cybersecurity of an organization’s critical infrastructure

The defense in depth approach is one of the most important principles of microservices security, as it creates multiple layers of security to prevent attacks. It includes such security measures as:

• Filtering communication flows
• Authenticating and authorizing access to microservices
• Using encryption technologies

Make sure to secure your internal environment from any external connections, as it is the first layer of defense. For example, check that you’re not using images from a public repository, as they may pose a security risk to your application. Administer the host via a known private network so there will be no public attack surface.

9. Secure access to microservices with API access control

APIs are the key to microservices applications. Software based on this technology has multiple independent API services that require additional tools.

Thus, API access control that ensures secure authentication and authorization is crucial for microservice security. Access to APIs that can process sensitive data should require authentication tokens that have either been digitally signed or verified with an authoritative source.

An OAuth/OAuth2 server is commonly used by developers and administrators to obtain tokens for accessing an application via an API. For security reasons, you should also apply transport layer security (TLS) encryption to protect all client–server communications.

10. Use container-native monitoring tools

Monitoring containers is essential, since it helps you to:

• Gain insights into container metrics and logs
• Understand what’s happening at the cluster and host level, as well as within the container
• Make better informed decisions, such as when to scale instances in or out, change instance types and purchasing options, etc.

To establish efficient monitoring, however, it’s best to use container-native monitoring tools. For example, when working with Docker, developers typically use Docker Security Scanner or other specifically designed tools to detect any potential threats to an application.

Monitoring tools first collect events and then examine them against security policies. A deterministic policy can define what services can be run and which containers are allowed to make external HTTP requests. A dynamic policy can create a baseline of normal communication activities and notify about traffic spikes or unusual traffic flows.

11. Use orchestration managers

Orchestration is a complex process that automates the deployment, management, scaling, and networking of microservices and containers. Usually, it can be implemented in two ways:

1. By using the API gateway as an orchestration layer
2. By coding orchestration as a separate microservice

Orchestrators pull images from registries, deploy those images to containers, and manage their running. The abstraction provided by an orchestrator allows you to specify how many containers are necessary to run a given image and what host resources need to be allocated for them.

By using an orchestration manager, you can not only automate the deployment of microservices but also ensure a certain level of security. For instance, orchestrators allow you to manage clusters of containers, segregate workloads, limit access to metadata, and collect logs.

Many orchestration managers also have built-in secrets management tools that allow developers to securely store and share secret data like API and SSL certificates, encryption keys, identity tokens, and passwords. There are many orchestration managers such as Kubernetes, Swarm, and Mesos as well as cloud-native management systems offered as part of Azure, Google Cloud Computing, and AWS.

Conclusion

A comprehensive security program for microservices-based software should address the entire application lifecycle. Using the described best practices, you can ensure the secure development and deployment of containers and microservices.

At Apriorit, we have dedicated teams with experience in cloud computing, cybersecurity projects, and security testing ready to help you boost microservices security capabilities as well as build high-quality and secure containerized solutions of any complexity.

Contact us to start securing the architecture of your microservices-based application!