Incomplete Guide for Securing Containerized Environment

(This article is a collection of best practices and tips regarding securing containerized environments. And there is a previously published article about the K8S hardening.)

And Understanding How Containers Present Unique Security Challenges

We all know that running cloud-based containers includes the flexibility of spinning applications up and down for users — or what we call “WORA (write once, run everywhere)” and better integration with DevOps. They also provide cost-saving infrastructure compared with monolithic applications running on servers or virtual machines.

Different companies, big or small, are looking into how to run their business applications with this technology. As a result, the adoption of containerized applications is at an all-time high. According to the survey from the Cloud Native Computing Foundation (CNCF):

Kubernetes adoption among the ever-expanding cloud-native community is approaching 100%, meaning those investing in cloud-native are strongly bought in and excited for the future.

The Container’s Threat Landscape

When we look into the recent data revealed by Shodan, 243,469 K8S clusters are publicly exposed. Furthermore, these Kubernetes clusters also exposed port 10250, used by the kubelet as a default setting. As a result, attackers could scan the internet and find those clusters with minimum effort.

Once the cluster is found, attackers could use the kubelet API as an entry point in targeting Kubernetes clusters to mine for cryptocurrency. More impactful actions are possible, such as lateral movement to other internal resources, privilege escalation, and finally, jumping into the jewel💎 — the confidential data.

With more and more data in containers, so does the interest in cyberattacks targeting such applications. According to VMware’s “State of Kubernetes survey 2022”, 97% of organizations have concerns about Kubernetes security.

Redhat also provides similar data with their “2022 state of Kubernetes security report”, saying that 94% of the respondents have experienced a security incident in their Kubernetes and container environments during the last 12 months.

Akamai’s security researcher Larry Cashdollar launched an exciting but alarming experiment by setting up a standalone Docker as honeypot Dock to see how frequent and what kind of attacks would be recorded. The result is scary. He specifically configured the honeypot simply to look like a vulnerable cloud instance.

Containers and Kubernetes attacks are common, and organizations have difficulties protecting themselves. Meanwhile, the businesses are transforming, and even though there are security concerns, they are putting more and more data into them.

3 Main Security Concerns in Containers

Although different, a container can still be compromised in many ways, just like traditional infrastructure. Therefore, I will highlight the most common container risk factor here.

1# Misconfiguration

Based on the prediction of Gartner from their analysis, by 2025, 99% of cloud security failures will be the mistakes of customers. As you can imagine, containers are often deployed in very dynamic environments. Therefore, any misconfiguration, no matter access, networking, or other related settings, can lead to a stealthy security hole for users but not to cybercriminals.

Basic misconfiguration accepts the default settings in the production environment, which are far less secure than customized, hardened settings. Attackers can find a problem in the default settings faster than understanding a new one. Smaller companies are generally using default configurations than large organizations.

Such small mistakes can lead to expensive problems. When we look back into Sep 2021, published by Palo Alto Networks, the Siloscape malware is the first known windows container malware and heavily obfuscated malware targeting K8S. According to the same research team, the primary purpose of Siloscape is to open a backdoor from poorly configured K8S clusters. As a result, the malware can further run malicious containers.

2# Vulnerable Container Images

Container images are static files with executable code that can be used to create a new container on a computing platform. Images are available via open-source repositories that are widely accessible online for easy download and deployment.

What if someone uploads the pre-installed malware into images and plants it into the public repository? When someone deployed those images, the malware would be running on day-0 without any action from the attackers. It’s a specific type of supply chain attack.

In 2020, a bug was found (CVE-2020–15157) in the Containerd runtime software for complete container lifecycle management. Adversaries could exploit it by creating specially-crafted container images to steal the host’s token, which they were pulled into a project. By the time they obtain the token, they can disguise themselves as the author and take hostage of a cloud project.

Another example of how attackers leverage malicious container images is illustrated by Team Nautilus, a threat research team from Aqua Security. They found five container images on Docker Hub used for mining cryptocurrency and as part of a supply chain attack specifically targeting cloud-native environments.

3# Zero-Day Vulnerabilities (+ Known CVEs)

Perhaps zero-day vulnerabilities are a big problem for all software — and containers are no exception. Although cloud providers invest heavily in securing their platforms, it’s unavoidable that zero-day vulnerability can exist and put customers at risk.

The most alarming example of this category is “Azurescape,” found by Palo Alto Networks. Similar to Siloscape, Azurescpe is the first known vulnerability that could allow a user of a public cloud service to “escape” the environment (Microsoft Azure).

This container escape is just an example of what known or zero-day vulnerabilities can impact a container environment. One public-facing hole could lead to a chain of exploits. When combined with a few misconfigurations, it would become a container takeover or even infiltrate other customers’ instances.

A report in February by Sysdig about cloud-native and security usage found that 75% of containers contain high severity vulnerabilities. However, the report also stressed that some respondents stated they are prepared to take these risks to move faster and release applications quicker.

Best Practices for Container Security

Containerized environments can introduce distinctive challenges for visibility and application of security controls, but following a layered-security approach can significantly reduce the attack surface.

The traditional approach of having a list of ‘bad’ things to look for won’t work in a container-based environment. Given the scalability and agility of these environments, organizations must be able to analyze the operational data in real-time and look for abnormal behaviors in runtime.

0# Continuous Improvement

The first thing for container security is implementing security practices in the CI/CD pipeline and the entire build, store and deploy process of container images that include:

• Static Application Security Testing (SAST);
• Securely storing the container images,
• Vulnerability scanning for container images and;
• Enforce image content trust.

Defense-in-Depth (DiD) in Containers

The container security best practices don’t just include the delivered applications and then securing the container’s image itself, but the entire stack for building, distributing and executing the container.

A layered approach should be followed. For starters, this includes:

• Cloud platform configuration
• The computing host (physical or virtual)
• The Operating System
• K8S cluster
• The container runtime
• Namespace
• The application code

Each layer would have its own security controls. For example, keeping the cluster infrastructure up to date is always important. But for the purpose of this guide, I would skip the part that is outside the K8S cluster.

1# Role-Based Access Control (RBAC)

RBAC is the frontline of defense in Kubernetes. It lets the user control who can access the API and their permissions. Official Kubernetes document provides detailed descriptions for Role-based Access Control (RBAC). Although RBAC is enabled by default, simply having RBAC enabled is not sufficient. Authorization policies should be handled carefully and properly applied.

Tips: Always use RBAC with the least privilege principle.

As a baseline, we need to make sure not to give cluster-wide permissions and not give anyone cluster-admin privileges unless necessary. That means using the RBAC policy to limit users and groups to only the actions and tasks they need. Also, confirm that the Kubernetes service and other user accounts have minimal privileges granted.

2# Benchmarking and Remediation

As stated above, misconfiguration is the primary source of container-related attacks. Thus, to spot those misconfigurations, we need something to compare with. The Center for Internet Security (CIS) is the de-facto standard among various sources of best practices, benchmarks, and hardening guides, the Center for Internet Security (CIS) is the de-facto standard.

Whether it is a physical server where the Linux is installed as a platform for containers, a virtual machine provisioned on a cloud service provider, or an on-prem box, it may contain insecure out-of-the-box settings that no one knows.

After all, we cannot assume that a cluster provisioned by a cloud provider is perfectly secured. Apart from K8S benchmarking, which is the most complex for cybersecurity professionals to review, CIS also provides similar checking for physical servers, especially Linux. CIS has a benchmark for Distribution Independent Linux and one specifically for Debian, Centos, Red Hat, and many other distributions.

3# Cloud Security Posture Management

Benchmarking can be extended to cloud provider accounts, too. It is different from the other settings as this part belongs to the management plane. Developers may not have access to those settings, but it is equally crucial for the overall security of a containerized environment.

A separate security control defined by Gartner for this area is called “Cloud Security Posture Management” (CSPM). Initially, CSPM supported you in discovering and visualizing the IaaS and PaaS asset inventory. But it may be extended to help you save time and minimize risk across your entire cloud/container security stack. According to Gartner, CSPM should include the capabilities of:

• Continuous visualization of configurations
• Assets discovery, classification, and risk assessment
• Standard misconfiguration remediation and logging
• Compliance reports with security frameworks (e.g., PCI DSS, HIPAA, SOC 2…)

4# IaC Scanning

IaC stands for Infrastructure-as-code, which refers to the technology and processes used to manage and provision infrastructure using code. Although IaC is not directly related to containers, it usually comes with DevOps. IaC simplifies DevOps processes such as peer reviews, version control, automation, and CI/CD delivery.

IaC comprises resource declarations, input and output variables, configurations, and parameters. IaC is usually JSON, YAML, or HCL based, and a single file can contain all resources and configurations required to spin up a whole infrastructure, including:

• compute,
• storage,
• networking,
• identity access management,
• security, and more.

Apart from its operational efficiencies, IaC can be employed to automate security processes and integrate them with DevOps as part of the transition to DevSecOps. However, IaC is a two-way sword.

By provisioning cloud resources across environments with a unified, standard language, developers and operations can better readily remain on the same page and work together to secure cloud-native applications. However, it also creates hidden traps inside the code without review.

IaC scanning software can parse standard cloud-native format templates. Then, they can set controls based on security best practices, providing developers or administrators with the knowledge of where they should apply further hardening to improve the security of the declared IaC environments.

These tools can separate security problems such as:

• a Docker image initiated to run as the root user,
• a Kubernetes manifest that requests a node’s privileged access to the file system, or
• an Ansible/ Terraform script that configures an open access S3 bucket.

As a result, adding security checks to IaC can isolate security vulnerabilities into a dedicated build and release pipeline. Likewise, IaC security simplifies workflows by embedding them into developer workflows to support cloud insight in both build and runtime.

Final Words

Although cloud-native technologies and containers are still a novice, the core security challenge remains; Organizations must consider how to accelerate development while retaining robust security controls, no matter it is a monolithic application or a containerized environment.

These two objectives are still competing for equal attention in the container space. Although having containers in the picture fostered scalability and agility, they also increased the overall complexity.

Securing containers needs a comprehensive approach. First, we must ensure that we address all teams’ needs within your organization. Next, make sure we can automate the procedure to fit the DevSecOps processes and that we also can meet deadlines and launch applications fast while protecting all of them.

Security can no longer be an enforcer at the last minute with demands to change your workflow. Instead, it needs to build trusted security controls and automated processes from the start to address security concerns and make it easier to bridge the gap between each team.

Thank you for reading. May InfoSec be with you🖖.