Kubernetes Container Security: DevSecOps Best Practices

Kubernetes has become a popular tool for deploying and managing containerized applications at scale. With more companies adopting automated CI/CD pipelines to deploy containerized applications into K8s, release cycles have become faster and deployments are done frequently. But at the same time challenges on security have risen. They cannot be overlooked in this fast-paced development environment.

The traditional approach to security where it is thought of only after deployment no longer works in DevOps. Additionally, container technology comes with its own security risks. For instance, the attack surface is wider in containers, as all the applications running share the same host OS. Infiltrating one container will jeopardize the security of all the other containers sharing the same host OS.

To mitigate any security risks with containers and Kubernetes clusters, the IT security teams need to adopt security best practices throughout the application development lifecycle. This is where DevSecOps principles come into the picture. In this blog, we will dive deeper into Kubernetes container security and DevSecOps best practices by exploring the following topics:

Policy and governance for Kubernetes application

Policies and governance refer to a set of rules and best practices followed by the development, security, and operations team to ensure the safety of containerized microservices. It also includes assuring the software delivery process complies with industry regulations and standards such as HIPAA, GDPR, PCI, etc.

Pod Security Policy (PSP) is an example of a policy for containerized applications and delivery processes. It is particularly used for Kubernetes applications and helps set rules for pod creation and management. Kubernetes also provides policy frameworks, such as Network Policies, to govern and manage the communication between pods.

Usually, the DevSecOps team implements policies and governance for container security in the software delivery or CI/CD process. They are responsible for ensuring that security is not an afterthought and is well integrated (refer to Fig A) while developing testing and deployments of containerized applications.

DevSecOps best practices for Kubernetes container security

Let us now look at some container security best practices or steps in the CI/CD process that the DevSecOps team can take to attain tighter security in Kubernetes. Please note the best practices do not represent as a pedagogy for ensuring Kubernetes security, but a bare minimum steps to improve security posture of containerised application significantly. Later towards the blog, we will also see how some of these can be implemented or enforced into CI/CD workflows to automate the security and compliance checks of containers.

Secure your container image creation

In a CI/CD process, the container is introduced right after a code undergoes a build process when a container image is created. Below are some security best practices to follow while creating a Docker image:

• Ensure the security of the base image: Every new container is built upon a pre-built, stable image called the base image. It is crucial to ensure that the base image used for creating a new image is the latest version, patched with the latest security updates. There are also tools such as Docker Scout which can be used to run security scan on the images.
• Use multistage builds to lower the image size: A multistage build uses multiple base images in a single Dockerfile. It helps to reduce the size of the new image as it will contain only the necessary components required for it to run, further reducing the attack surface.

Below example uses two different base images: golang:1.15-alpine and alpine:3.12. The first image (Golang) builds the application, and the second image (alpine) runs the application. The binary file from the first stage is copied to the second stage, keeping only the necessary components to run the app and reducing the image size and attack surface.

# Stage 1: Build
FROM golang:1.15-alpine as builder

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .

# Stage 2: Run
FROM alpine:3.12

RUN apk --no-cache add ca-certificates

WORKDIR /root/

COPY --from=builder /app/main .

CMD ["./main"]

• Expose only the required ports: Exposing unnecessary ports can allow hackers to get into a cluster and steal sensitive data. It is always better to only expose the minimum required ports while (1) specifying the container ports using the expose command and (2) while opening a container to the outside world using Service YAML in the Kubernetes environment.

Automatically pull container images from registered sources

Apart from creating container images, it is common to pull already created images from a registry like Dockerhub. Organizations mostly have their own private registry to store their internal images safely and securely.

Some security best practices while pulling images are:

• Pull from trustworthy sources: Ensure that images are from a trustworthy source and verified. Additionally, make sure that the container image is signed by the publisher. The image should be the official one from the vendor.
• Use the latest version: This ensures that the image is up-to-date with the latest security patches.
• Vulnerability scanning: Use tools like Aqua Trivy, Clair, etc. to scan the images for Common Vulnerabilities and Exposures (CVEs).

Note that image vulnerability scanning is not a one-off process. If an organization’s CI pipeline stores the images in a private registry, they should be scanned continuously and automatically to alert them of known and new vulnerabilities.

Ensure container security during the deployment

Once a container image is ready and deployed on a host OS, the DevSecOps team has to make sure that the host OS and container runtime are secured. If the containers are running in a Kubernetes cluster, additional security measures must be taken.

Isolating container from host OS

• Isolation between containers and host: Use network segmentation by isolating the containers from the host. It ensures that the host file system is not accessible through containers, which prevents unauthorized access to sensitive data.

There are other ways to provide further isolation between the host and containers:

• Separate user namespaces: User namespaces offered by Linux APIs help to isolate specific aspects of the OS. Different namespaces for containers and host OS means they will not see each other's processes, as user IDs will differ.

The below sample shows the container and host OS in different namespaces or with different user IDs (1000 and 1001).

# Start a new container in a new namespace
$ docker run -it --rm --name mycontainer --network none --ipc none --uts none --pid none --user 1000:1000 ubuntu:latest

# Open a shell inside the container
root@9c7f6d0c6b7f:/# 

# Check the user ID of the current user
root@9c7f6d0c6b7f:/# id -u
1000

# Exit the container
root@9c7f6d0c6b7f:/# exit

# Check the user ID of the current user on host
$ id -u
1001

• Container sandboxing: Sandboxed containers create virtualized environments by adding a new runtime, which then starts containers inside them. Kata container is an open-source container sandboxing project that provides a secure container runtime with lightweight VMs.
• Security Enhanced Linux (SELinux) and AppArmor: Integrating security modules like SELinux and AppArmor for the Linux kernel help to implement policies that limit containers’ access to the host’s file system. They can also limit the actions a container can perform on the host.
• Use a minimal host OS: Ensure that the host OS contains only the necessary components and libraries required to run the containers. And frequently update the OS to get the latest security patches.

Ensure container runtime security

The container runtime is the software installed on the host OS that helps it run and manage containers. In a Kubernetes environment, a container runtime is present in all pods.

• Enforce security policies: The policies can include restrictions on which containers should run in the cluster. This also includes using the least privilege principle, which allows only the required privilege for the containers to run.
• Run containers as users, not root: Containers should ideally run as users and in read-only mode. It prevents attackers from modifying the container file system by writing malicious code.
• Limit access to container runtime daemon and APIs: Prevent access to the underlying system that container runtime is running on using access controls such as authentication and authorization mechanisms.

Apart from implementing the above security practices, it is crucial to set up a logging solution that logs data of container runtime and the applications running in the containers. This helps identify security issues like unauthorized access and comes in handy while troubleshooting security incidents.

Enforce Kubernetes environment security

• Role-based access control (RBAC): Manage users and permissions to access Kubernetes API using RBAC, and keep the privileges limited. Roles can be set on namespace or cluster-wide.
• Network policies: Set network policies in K8s to limit and manage communication between pods in a cluster. This can be done by configuring the NetworkPolicy resource. Configuring network policies on a service-to-service level would require a service mesh like Istio.
• Manage secrets: Secure and encrypt Kubernetes Secrets object, which stores sensitive data like credentials, tokens, and keys. Let only the necessary pods and services access it and change or rotate secrets frequently. Using an external secret store such as Azure vault is recommended.
• Secure etcd store: etcd store manages sensitive data such as configuration data, secrets, state data, and other metadata for Kubernetes. Use firewall rules and TLS protocols to secure communication between etcd clients and services.
• Admission controller plugin: An admission controller in K8s checks requests to the Kubernetes API server and allows or denies the requests based on a set of rules. They govern and enforce the use of a cluster. Pod Security Policy (PSP) is an admission controller that enforces security policies on pods. PSPs ensure that only the pods that are secure and compliant run in the cluster.

It is also important to track all access to the Kubernetes control plane. Enabling access logs for all API requests will help identify unauthorized access attempts. Additionally, it is a good practice to continuously scan Kubernetes manifest files to ensure security best practices and compliance before they are deployed in production.

Kubernetes security using open source Devtron CD platform

Challenges are plenty with securing containerized applications and Kubernetes environments. Throughout the container lifecycle, from creation to deployment in a Kubernetes cluster, we have seen three key areas crucial in every stage: authorization, security scanning, and security policies.

However, it is tedious to set up and configure multiple tools for these security purposes. This is where open-source Kubernetes-native continuous delivery(CD) tools like Devtron can be a valuable asset for the DevSecOps teams. Devtron offers a pull and push-based CI/CD pipeline for deploying and delivering Kubernetes applications frequently, safely, and securely. And DevSecops team can enforce security and compliance policies into Devtron's CI/CD workflows.

Here is how Devtron seamlessly integrates security into its CI/CD pipeline:

• Authorization: Devtron helps in managing authentication and authorizing access to resources. It is easier for the DevSecOps team to set a robust SSO with role-based access, which can be added to a user via user permissions, permission groups, and API tokens to restrict use action on Kubernetes resources, and Helm charts.
  Read more about authorization here: https://docs.devtron.ai/global-configurations/authorization
• Security scanning: Devtron integrates security scanning tools such as Trivy and SonarQube into its CI/CD pipeline to scan and find vulnerabilities in container images before they are considered for deployment. In the case of true-positives, a pipeline will stop and fail the deployment process. With automated security checks of deployment, DevSecOps can facilitate the security checks of all the deployments at scale.
  

Read more about vulnerability scanning here: https://docs.devtron.ai/usage/integrations/clair

• Custom Security Policies for Kubernetes deployment: Users can set policies on Devtron based on the level of vulnerability—critical, moderate, and low. This feature allows users to block or whitelist any vulnerabilities. For e.g. you can create a BalckOutWindow compliance policy to ensure no applications are getting deployed during business peak hours. Or create policies to avoid developers deploying their apps directly into production without testers or project managers' approvals.

The best part is Devtron allows you to create these security policies and set them at different levels, such as global, cluster, user, environment, or application.

Read more about security policies here: https://docs.devtron.ai/usage/security-features