Securing the 4 Pillars of Kubernetes CI/CD with Devtron

Speed fuels the current software delivery process. Whenever there is new business logic, developers have this urge to instantly get it into the hands of end users. Agile methodologies and DevOps principles support this approach: roll out incremental changes as fast and as frequently as possible.

The reason security becomes the last checkpoint or often the overlooked part of the process is because of the fear that it would slow everything down. However, some organizations paid a heavy price for this attitude.

Back in 2019, an attacker exploited Kubernetes misconfiguration in the system of Capital One — one of the largest banks in the US. A misconfigured firewall gave the attacker access to AWS’ IAM role. She stole over 80,000 bank account numbers, more than 140,000 Social Security numbers, over 1 million Canadian social insurance numbers, and millions of credit card applications. The bank reported that the breach might cost it over $300 million. (Read about it here: Great year for hackers: Top 2019 data breaches so far.)

So, how do organizations find a balance between speed and security in the delivery process? One logical solution would be to automate possible security checks in the pipeline and ensure DevSecOps principles. But the real question is, where does one start with securing Kubernetes in CI/CD?

To answer that, we have explained the four pillars for securing Kubernetes in CI/CD and some best practices to be followed under each pillar. Let us explore them one by one.

Authentication and Authorization (Authn/z)

Securing CI/CD pipelines

A strong authentication and access control are the first steps towards securing a Kubernetes CI/CD pipeline. It starts with enforcing policies around what the pipeline can access, who has access to it, and what permission it has. And it should be led by implementing the least privilege policy, which means granting only the required minimum privileges for the pipeline components to perform their tasks.

The downside of not maintaining proper access control is a compromised pipeline. For example, if an attacker gains access to a pipeline component with full privileges, they can easily push malicious code or such artifacts to production, especially if there are no strict reviews or approval processes. Some best practices to not let this happen are:

• Empower users with the least privileges necessary to execute, view, or manage CI/CD pipelines. This can be done by access controls through configuring identity and access management (IAM) and role-based access control (RBAC) standards.

• Avoid providing pipeline components with administrative access over the cloud

• Introduce a manual review and approval stage in a CI/CD pipeline before a code is sent for builds or deployments are triggered.

• For open-source CI/CD pipeline providers such as Jenkins, monitor and verify their third-party plugins before they are used in the pipeline. Avoid plugins that are no longer supported. For example, Docker Hub plugins supporting V1 API endpoints are no longer necessary.

Securing Kubernetes resources with Authn/z

There are different authentication and authorization methods to prevent unwanted access to a K8s cluster. For authentication, there are client certificates and ServiceAccount. A client certificate is used to authorize human users, and service accounts authorize applications. In any case, it is always a good practice to limit the approved authorization to specific namespaces using RBAC Roles and RoleBinding. For users or apps who need cluster-wide permissions, ClusterRole and ClusterRoleBinding can be configured.

To simplify access management and have a good view of who has access to what, it is better to use open-source tools like Devtron Kubernetes Dashboard. The dashboard provides users with a centralized panel for access management (refer to Fig A below), which helps in setting resource-specific admin or view access without manually configuring RBAC.

In a Kubernetes cluster, it is not advisable to run privileged containers, since they have full access to the host system and sensitive resources. Effective policies should be in place to fail deployment in case of a privileged container. Additionally, it is better to put allowPrivilegeEscalation: false under securityContext in the pod manifest file to prevent it from accessing other containers and namespaces. Containers should not run as root, but as a service user with a read-only root file system and specified cluster resource limits set for them. All these measures prevent unauthorized access to the cluster and reduce the attack surface in case of a security breach.

Secrets management to handle passwords and tokens

Secrets management for CI/CD pipeline

Secrets such as tokens, API keys, SSH keys, etc. are spread over the entire pipeline in a CI/CD workflow. From credentials to access the Git repository, CI servers, and cloud providers, to monitoring and collaboration tools, secrets are needed to authenticate access and perform various roles. This makes securely storing and retrieving credentials for various CI/CD components a challenging endeavor.

Proper secrets management is one of the core policies to ensure the security of CI/CD pipelines, as they make sure the secrets are encrypted and not exposed. Some secrets management best practices are:

• Do not use the same secrets for the same process in different environments (same database credentials in dev, test, or prod, for example)

• Do not hard-code secrets or store them in git repositories

• Store secrets in environment variables and use them to jobs as they are running as part of the pipeline

• Store secrets outside the pipeline in secret management tools such as HashiCorp Vault.

• Regularly rotate secrets

Kubernetes secrets management

In a K8s cluster, the Secrets manifest file stores sensitive data like passwords, tokens, etc. They are not encrypted by default, but only base64 encoded. The secrets along with the cluster data are stored in plain text in the etcd store. Thus, it is advisable to restrict access to etcd data and encrypt etcd backups. Managing etcd outside the cluster will prevent it from being exposed if an attacker gains access to the cluster.

Another way to secure K8s Secrets is to encrypt them into sealed secrets. This form of cryptography encryption works based on public key encryption and generates two keys — a public key and a private key. One key is used for encrypting the secrets, and only the other key can decrypt it. That means, once the secrets are encrypted, it can only be decrypted by the sealed secrets controller running in the target cluster.

As mentioned above, storing secrets are environment variables or in a data volume provides additional security by isolating them from the code or pipeline configuration. Devtron provides options for both while configuring Secrets (refer to Fig B).

Using secret management tools like HashiCorp Vault will further help to secure K8s secrets, as it acts as a central system to store and access secrets securely. Vault provides encryption to Secrets in transit and at rest using TLS encryption and AES 256-bit CBC encryption, respectively. Additionally, Vault allows dynamic secrets and updates the pods that use the secret while rotating them. Following all these best practices with a secrets management tool will ensure that secrets are stored and managed securely.

Scanning and testing for security vulnerabilities

Testing in CI/CD pipelines

One of the major benefits of CI/CD pipelines is that developers can incorporate multiple scanning and security tests into the workflow. These continuous tests will look for any known vulnerabilities in the code and also for any new CVEs that are discovered after deploying the application. They will ensure that the application is secure as long as it is running in an environment.

Two types of software testing that should be implemented in CI/CD pipelines to identify security vulnerabilities are SAST (Static Application Security Testing) and DAST (Dynamic Application Security Testing).

• SAST is an open-box testing method that scans the application code inside out for security vulnerabilities, like the OWASP top 10 security risks. It scans the application code at rest, and it is done during the development phase in the pipeline.

• DAST tests the application after it is deployed and running. It is a closed-box testing method and does not access the application source code. The goal of DAST is to observe the application’s runtime behavior and analyze if it is susceptible to cybersecurity threats.

Apart from testing the artifact, the provisioned infrastructure should also be scanned for possible misconfigurations and to identify any vulnerabilities in underlying resources. Since infrastructure as code (IaC) is the norm, SAST and DAST should be extended to IaC configuration files and codebase, and they should be embedded into the CI/CD pipeline.

Scanning in K8s cluster

In a K8s cluster, the base image and dependencies of a container image should be scanned before deploying it. This should not be a one-off process and should be automatically done multiple times in the pipeline. That is, images should be scanned before building, after they are stored in a container registry, and also during runtime. Using security tools like Snyk will help to scan container image dependencies at the application level and the OS level.
Devtron integrates Clair — an open-source project for the static analysis of vulnerabilities in application containers — to its platform and provides image vulnerability scanning out of the box (refer to Fig. C below). It helps to see all the packages a container image has and identify if they have any vulnerabilities, which can be resolved before deploying the image into a cluster.

Along with scanning and testing container images, it is a good practice to test the cluster against the CIS (Center for Internet Security) Kubernetes Benchmark. The CIS Kubernetes Benchmark is a comprehensive set of secure configuration guidelines for Kubernetes, created through a collaborative effort of the community. The benchmark test can be carried out by deploying an open-source security tool called Kube-bench. This will ensure cluster security and reduce the risk of security incidents.

Logging and monitoring for better visibility

Building inventory and monitoring CI/CD pipelines

Any security issues in the CI/CD pipeline should be immediately attended to because the interconnected nature of pipelines gives attackers a high attack surface. Logging and having visibility into all the activities in the K8s CI/CD pipeline thus become crucial for DevSecOps, especially for incident response in case of a security breach. This helps to reduce the potential impact of a security breach and ensure that the pipeline remains secure.

There are four crucial elements to achieve sufficient logging and visibility in a K8s CI/CD pipeline:

• Build an inventory of all the systems in use within an organization: A security breach could be in any pipeline stage. It could be in the git repo, CI pipeline, container registries, etc. This is why creating an inventory of different systems used in the organization is the starting point.

• Logging: Enable logs once relevant systems with potential threats are identified.

• Centralized location for logs: Sending logs to a centralized location, such as a Security Information and Event Management (SIEM) system, can help aggregate and correlate logs across different systems for improved detection and investigation capabilities.

• Set alarms: Create alerts based on security parameters, like resource consumption or abnormalities.

Using monitoring tools like Prometheus will make the job easier as it will collect metrics from configured targets at regular intervals, and it is easy to build Grafana-style dashboards on top of it for better visualization.

Monitor communication between services across K8s clusters

While deploying into K8s, it is important to have full-stack visibility into the cluster to prevent any runtime attacks. Proper logging and monitoring tools should be in place to get alerts on abnormal K8s resource consumption, or any abnormal behavior in the cluster. One such logging tool is Fluent Bit, which processes container logs from the file system and centralizes them in third-party storage services like Elasticsearch, InfluxDB, etc.

Another critical aspect to monitor in a Kubernetes cluster is the communication between services. Using a service mesh like Istio provides visibility into which services access which endpoints in a cluster, making it an ideal solution for monitoring communication between services in a Kubernetes environment. Besides, Istio provides secure communication between services by providing additional features like mTLS encryption.

By having visibility into the cluster resources and traffic between the services across clusters and clouds, it becomes possible to proactively identify and address any potential security threats.

Kubernetes CI/CD pipeline security with open-source Devtron platform

A CI/CD pipeline deploying into Kubernetes is optimized for speed and scalability. In such a rapid continuous delivery system, developers are mainly concerned about how fast the code changes are made into production, and security can become an afterthought.

But it should not be the case anymore. Because attackers are also aware of the new wave of change in SDLC and shifting their attention to CI/CD. They are aware of the complexity of securing Kubernetes CI/CD pipelines and adapting their techniques to explore vulnerabilities in them. This is why organizations should use tools like the Devtron platform, as securing K8s CI/CD pipelines is one of its core goals. Some security features offered by the Devtron platform are:

• Multi-level security policy at global, cluster, environment, and application levels (refer to the below image) for efficient hierarchical policy management

• Define policies and exceptions for Kubernetes resources

• Advanced workflow policies like blackout window and branch environment relationship to secure build and deployment pipelines

The best part about all these features is that everything from security to governance is tucked inside and can be configured from a dashboard with a user-friendly UI. Feel free to log in with GitHub and try the open-source Devtron platform here: https://preview.devtron.ai/dashboard/login/

References and additional readings:
Cider Security -  Top 10 CI/CD Security Risks  (https://www.cidersecurity.io/top-10-cicd-security-risks/)

Devtron - Kubernetes Container Security: DevSecOps Best Practices (https://devtron.ai/blog/kubernetes-container-security-devsecops-best-practices/)