Securing the 4 Pillars of Kubernetes CI/CD with Devtron

Introduction

Modern development thrives on rapid iteration, deploying features quickly and frequently. But when speed takes priority over safety, security becomes an afterthought.

The 2019 Capital One data breach caused by a Kubernetes misconfiguration exposed hundreds of thousands of sensitive records and cost the company over $300 million. It’s a stark reminder that in cloud-native systems, even small security gaps can lead to catastrophic exposure.

The challenge isn’t choosing between speed and security — it’s achieving both through DevSecOps, where security is automated and embedded directly into your CI/CD workflows.

Let’s look at the four pillars of securing Kubernetes CI/CD pipelines and how Devtron simplifies each one through unified visibility and governance.

Authentication and Authorization (Authn/z)

Securing CI/CD pipelines

A strong authentication and access control are the first steps towards securing a Kubernetes CI/CD pipeline. It starts with enforcing policies around what the pipeline can access, who has access to it, and what permission it has. And it should be led by implementing the least privilege policy, which means granting only the required minimum privileges for the pipeline components to perform their tasks.

The downside of not maintaining proper access control is a compromised pipeline. For example, if an attacker gains access to a pipeline component with full privileges, they can easily push malicious code or such artifacts to production, especially if there are no strict reviews or approval processes. Some best practices to not let this happen are:

• Empower users with the least privileges necessary to execute, view, or manage CI/CD pipelines. This can be done by access controls through configuring identity and access management (IAM) and role-based access control (RBAC) standards.

• Avoid providing pipeline components with administrative access over the cloud

• Introduce a manual review and approval stage in a CI/CD pipeline before a code is sent for builds or deployments are triggered.

• For open-source CI/CD pipeline providers such as Jenkins, monitor and verify their third-party plugins before they are used in the pipeline. Avoid plugins that are no longer supported. For example, Docker Hub plugins supporting V1 API endpoints are no longer necessary.

Securing Kubernetes resources with Authn/z

There are different authentication and authorization methods to prevent unwanted access to a K8s cluster. For authentication, there are client certificates and ServiceAccount. A client certificate is used to authorize human users, and service accounts authorize applications. In any case, it is always a good practice to limit the approved authorization to specific namespaces using RBAC Roles and RoleBinding. For users or apps who need cluster-wide permissions, ClusterRole and ClusterRoleBinding can be configured.

To simplify access management and have a good view of who has access to what, it is better to use open-source tools like Devtron Kubernetes Dashboard. The dashboard provides users with a centralized panel for access management (refer to Fig A below), which helps in setting resource-specific admin or view access without manually configuring RBAC.

In a Kubernetes cluster, it is not advisable to run privileged containers, since they have full access to the host system and sensitive resources. Effective policies should be in place to fail deployment in case of a privileged container. Additionally, it is better to put allowPrivilegeEscalation: false under securityContext in the pod manifest file to prevent it from accessing other containers and namespaces. Containers should not run as root, but as a service user with a read-only root file system and specified cluster resource limits set for them. All these measures prevent unauthorized access to the cluster and reduce the attack surface in case of a security breach.

Secrets management to handle passwords and tokens

Secrets management for CI/CD pipeline

Secrets such as tokens, API keys, SSH keys, etc. are spread over the entire pipeline in a CI/CD workflow. From credentials to access the Git repository, CI servers, and cloud providers, to monitoring and collaboration tools, secrets are needed to authenticate access and perform various roles. This makes securely storing and retrieving credentials for various CI/CD components a challenging endeavor.

Proper secrets management is one of the core policies to ensure the security of CI/CD pipelines, as they make sure the secrets are encrypted and not exposed. Some secrets management best practices are:

• Do not use the same secrets for the same process in different environments (same database credentials in dev, test, or prod, for example)

• Do not hard-code secrets or store them in git repositories

• Store secrets in environment variables and use them to jobs as they are running as part of the pipeline

• Store secrets outside the pipeline in secret management tools such as HashiCorp Vault.

• Regularly rotate secrets

Kubernetes secrets management

In a K8s cluster, the Secrets manifest file stores sensitive data like passwords, tokens, etc. They are not encrypted by default, but only base64 encoded. The secrets along with the cluster data are stored in plain text in the etcd store. Thus, it is advisable to restrict access to etcd data and encrypt etcd backups. Managing etcd outside the cluster will prevent it from being exposed if an attacker gains access to the cluster.

Another way to secure K8s Secrets is to encrypt them into sealed secrets. This form of cryptography encryption works based on public key encryption and generates two keys — a public key and a private key. One key is used for encrypting the secrets, and only the other key can decrypt it. That means, once the secrets are encrypted, it can only be decrypted by the sealed secrets controller running in the target cluster.

As mentioned above, storing secrets are environment variables or in a data volume provides additional security by isolating them from the code or pipeline configuration. Devtron provides options for both while configuring Secrets (refer to Fig B).

Using secret management tools like HashiCorp Vault will further help to secure K8s secrets, as it acts as a central system to store and access secrets securely. Vault provides encryption to Secrets in transit and at rest using TLS encryption and AES 256-bit CBC encryption, respectively. Additionally, Vault allows dynamic secrets and updates the pods that use the secret while rotating them. Following all these best practices with a secrets management tool will ensure that secrets are stored and managed securely.

Scanning and testing for security vulnerabilities

Testing in CI/CD pipelines

One of the major benefits of CI/CD pipelines is that developers can incorporate multiple scanning and security tests into the workflow. These continuous tests will look for any known vulnerabilities in the code and also for any new CVEs that are discovered after deploying the application. They will ensure that the application is secure as long as it is running in an environment.

Two types of software testing that should be implemented in CI/CD pipelines to identify security vulnerabilities are SAST (Static Application Security Testing) and DAST (Dynamic Application Security Testing).

• SAST is an open-box testing method that scans the application code inside out for security vulnerabilities, like the OWASP top 10 security risks. It scans the application code at rest, and it is done during the development phase in the pipeline.

• DAST tests the application after it is deployed and running. It is a closed-box testing method and does not access the application source code. The goal of DAST is to observe the application’s runtime behavior and analyze if it is susceptible to cybersecurity threats.

Apart from testing the artifact, the provisioned infrastructure should also be scanned for possible misconfigurations and to identify any vulnerabilities in underlying resources. Since infrastructure as code (IaC) is the norm, SAST and DAST should be extended to IaC configuration files and codebase, and they should be embedded into the CI/CD pipeline.

Scanning in K8s cluster

In a K8s cluster, the base image and dependencies of a container image should be scanned before deploying it. This should not be a one-off process and should be automatically done multiple times in the pipeline. That is, images should be scanned before building, after they are stored in a container registry, and also during runtime. Using security tools like Snyk will help to scan container image dependencies at the application level and the OS level.
Devtron integrates Clair — an open-source project for the static analysis of vulnerabilities in application containers — to its platform and provides image vulnerability scanning out of the box (refer to Fig. C below). It helps to see all the packages a container image has and identify if they have any vulnerabilities, which can be resolved before deploying the image into a cluster.

Along with scanning and testing container images, it is a good practice to test the cluster against the CIS (Center for Internet Security) Kubernetes Benchmark. The CIS Kubernetes Benchmark is a comprehensive set of secure configuration guidelines for Kubernetes, created through a collaborative effort of the community. The benchmark test can be carried out by deploying an open-source security tool called Kube-bench. This will ensure cluster security and reduce the risk of security incidents.

Logging and monitoring for better visibility

Building inventory and monitoring CI/CD pipelines

Any security issues in the CI/CD pipeline should be immediately attended to because the interconnected nature of pipelines gives attackers a high attack surface. Logging and having visibility into all the activities in the K8s CI/CD pipeline thus become crucial for DevSecOps, especially for incident response in case of a security breach. This helps to reduce the potential impact of a security breach and ensure that the pipeline remains secure.

There are four crucial elements to achieve sufficient logging and visibility in a K8s CI/CD pipeline:

• Build an inventory of all the systems in use within an organization: A security breach could be in any pipeline stage. It could be in the git repo, CI pipeline, container registries, etc. This is why creating an inventory of different systems used in the organization is the starting point.

• Logging: Enable logs once relevant systems with potential threats are identified.

• Centralized location for logs: Sending logs to a centralized location, such as a Security Information and Event Management (SIEM) system, can help aggregate and correlate logs across different systems for improved detection and investigation capabilities.

• Set alarms: Create alerts based on security parameters, like resource consumption or abnormalities.

Using monitoring tools like Prometheus will make the job easier as it will collect metrics from configured targets at regular intervals, and it is easy to build Grafana-style dashboards on top of it for better visualization.

Monitor communication between services across K8s clusters

While deploying into K8s, it is important to have full-stack visibility into the cluster to prevent any runtime attacks. Proper logging and monitoring tools should be in place to get alerts on abnormal K8s resource consumption, or any abnormal behavior in the cluster. One such logging tool is Fluent Bit, which processes container logs from the file system and centralizes them in third-party storage services like Elasticsearch, InfluxDB, etc.

Another critical aspect to monitor in a Kubernetes cluster is the communication between services. Using a service mesh like Istio provides visibility into which services access which endpoints in a cluster, making it an ideal solution for monitoring communication between services in a Kubernetes environment. Besides, Istio provides secure communication between services by providing additional features like mTLS encryption.

By having visibility into the cluster resources and traffic between the services across clusters and clouds, it becomes possible to proactively identify and address any potential security threats.

Securing Kubernetes CI/CD with Devtron

Traditional approaches to Kubernetes security rely on multiple disconnected tools, one for secrets, another for policy, and another for scanning. This fragmentation creates operational blind spots and increases risk.

Devtron eliminates this complexity through unified visibility, governance, and control, consolidating all security and compliance operations within a single Kubernetes-native platform.

Some security features offered by the Devtron platform are:

• Multi-Level Security Policies: Enforce hierarchical policies at global, cluster, environment, and application levels, ensuring granular governance and simplified compliance management

• Policy-as-Code Enforcement: Define, apply, and monitor policies that govern deployments, infrastructure, and runtime operations across all environments.

• Automated Compliance Scanning: Continuously validate configurations against industry benchmarks like CIS and ensure adherence to security standards.

• RBAC with Break-Glass Procedures: Maintain least privilege while enabling emergency access with complete auditability.

• Integrations with Secrets Management Tools: Seamlessly connect with Vault, AWS Secrets Manager, and other enterprise solutions for centralized credential management.

With Devtron, security, governance, and operational control converge; enabling teams to innovate faster while staying compliant and resilient.

References and additional readings:
Cider Security - Top 10 CI/CD Security Risks (https://www.cidersecurity.io/top-10-cicd-security-risks/)

Devtron - Kubernetes Container Security: DevSecOps Best Practices (https://devtron.ai/blog/kubernetes-container-security-devsecops-best-practices/)

Conclusion

Security shouldn’t be a barrier to speed; it should enable it.
By embedding policy-as-code, automated compliance, RBAC governance, and multi-level security policies directly into Kubernetes operations, Devtron empowers teams to deliver faster with confidence.

With unified visibility and governance built into every layer, Devtron transforms Kubernetes security from a patchwork process into a scalable, streamlined practice.

FAQs

Jyoti Sahoo

Jyoti crafts engaging content for business and tech leaders. He has led AI projects for EU governments and ANZ clients. A 3D printing enthusiast and solar energy advocate.

Prakarsh

Prakarsh is Head of DevOps at Devtron Inc, with 15+ years in the DevOps ecosystem. He drives automation, cloud adoption & security, optimizing processes for scalable, production-ready solutions.

Co-authors: Jyoti Sahoo, Prakarsh,