The Gatekeeper: Why Approval-Based Deployments are Essential for Production Environments on Kubernetes

In the fast-paced world of software development, speed and efficiency are paramount. However, when it comes to deploying applications to production environments, particularly on Kubernetes, it’s crucial to prioritize control and security. One effective way to achieve this is through approval-based deployments. In this blog post, we’ll explore the importance of this practice and how it helps maintain the integrity and reliability of production systems.

What is Approval-Based Deployment?

Approval-based deployment is a process where any changes or updates to the production environment must be reviewed and approved by authorized personnel before they are applied. This step acts as a safeguard, ensuring that only vetted and tested code makes its way into the live environment where it can impact end-users. Approval can be made at two different levels, i.e., approval for the configurations and approval for the container image.

The Critical Role of Kubernetes in Modern Deployments

Kubernetes has revolutionized how we deploy, scale, and manage containerized applications. Its flexibility and power make it a favorite among DevOps teams. However, with great power comes great responsibility. Kubernetes allows rapid changes and scaling, but without proper oversight, it can lead to significant risks, including downtime, security vulnerabilities, and performance issues. The adoption of Kubernetes has been increased from 71% to 89% as per CNCF 2023 Annual Survey and out of 89%, 18% are still evaluating Kubernetes to use in production. Though it’s known for its ability to orchestrate a container, Kubernetes does come with a lot of complexities, and using the right set of tools to create an abstraction is very important.

Why Approval-Based Deployments Matter

Approval-based deployments are important for any critical environment. Here are some of the reasons why approval-based deployments matter:

Enhanced Security

• Prevent Unauthorized Changes: Approval-based deployments ensure that only authorized individuals can make changes to the production environment. This prevents malicious or accidental modifications that could compromise the production environment.
• Review for Vulnerabilities: Each deployment undergoes a thorough review, reducing the risk of introducing security vulnerabilities. This is especially important for Kubernetes environments, where misconfigurations and vulnerable images can lead to severe security breaches.

Improved Stability

• Quality Assurance: By requiring approvals, teams can ensure that only stable, tested code reaches production. This minimizes the chances of deploying buggy or unstable applications, which can lead to downtime and user dissatisfaction.
• Controlled Rollouts: Approvals allow for controlled, staged rollouts, ensuring that any issues can be quickly identified and addressed without impacting the entire user base.

Compliance and Auditing

• Regulatory Compliance: Many industries, primarily Fintech organizations like Banks, are subject to strict regulatory requirements. Approval-based deployments provide an auditable trail of changes, making it easier to demonstrate compliance with these regulations.
• Accountability: The approval process holds individuals accountable for the changes they authorize. This promotes a culture of responsibility and thoroughness within the development and operations teams.

Operational Efficiency

• Preventing Errors: By catching potential issues before they reach production, teams can avoid costly and time-consuming rollbacks and fixes. This keeps the system running smoothly and efficiently.
• Continuous Improvement: The approval process often includes a review and feedback loop, fostering continuous improvement in the deployment process and overall code quality.

Implementing Approval-Based Deployments on Kubernetes

To effectively implement approval-based deployments in a Kubernetes environment, consider the following best practices:

• Define Clear Approval Policies: Establish users who can approve deployments with relevant permissions and clear distinction between the user who can approve the container images and critical environment configurations.
• Automate the Approval Workflow: Using tools like Devtron to automate the deployment pipeline, integrating approval gates that require manual sign-off at critical environments like production within the workflow.
• Monitor and Audit: Implement monitoring and auditing solutions to track changes and approvals. Regularly review these changes to ensure compliance and identify areas for improvement and easy rollback strategies if anything goes wrong with the latest releases.
• Continuous Training: Ensure that all team members are trained on the approval process and understand the importance of their role in maintaining production integrity.

Approval-Based Workflows with Devtron

Devtron streamlines the platform engineering workflows for Kubernetes. It provides an intuitive dashboard to deploy and manage all your workloads on Kubernetes through Kubernetes-native CI/CD pipelines as well as Helm Charts, the K8s package manager. With the help of Devtron, users can easily create approval checks for the configurations as well as for the container images for critical environments. Let’s dive into the approval-based workflows with Devtron.

Step 1: Devtron comes with granular RBAC which allows you to define different roles for users, who can approve the configurations and container images before it get deployed in critical environments.

Step 2: After the access has been granted, users can easily create automated workflows with Devtron and integrate the approval checks for configurations and container images. Creating workflows with Devtron is pretty straightforward just with a few clicks and you can create any type of workflows with Devtron, be it sequential or parallel as you can see in the below image.

Additionally adding an approval check is also pretty much straightforward. For the deployment pipeline of any critical environment i.e., prod, simply toggle on the approval checks and define the number of approvals required before you can deploy it. There are no custom scripting or external integrations are required as you can in the below image.

Devtron also comes with Protect Configurations which allows you to protect the configurations of specific environments. For any workflow that you have created, Devtron automatically created Environment Overrides for you as you can see in the [Fig. 3], which ensures the isolation of configurations for multiple deployment environments. To protect the configuration of any environment, simply toggle on the button as you can see in the below image.

If any changes are made in that environment, they first need to be approved by the respective approver, then only those changes can be deployed in the respective environment. Additionally, you can see a diff in the changes made by the user along with the user details. The user asking for approval can also write comments with details on why the changes have been made to provide a proper context to the approver as you can in the below image.

Step 3: After the workflow has been created, and configurations are done, the user can directly move to the Build & Deploy section where the user can trigger the pipelines. If it is an automated pipeline, as soon as you make a new commit, the pipeline will be automatically triggered, which would create a container image, deploy it into the utils environment, perform automated test cases if there are any in the pre/post deployment stages, deploy the same image to the stage environment and before it gets deployed into production, approval is required. To raise an approval request, click on the approval check button as described in [Fig. 3], there you can see all images and additionally, you can also give Image labels and Comment for the change made with this image as you can see in the below image.

The user can select any of the people from the list to approve their container image. The user who is requested will also be notified through email and can approve it via email, additionally, the user can also see the Approval Pending in the dashboard itself with the details of the user who raised an approval request along with labels and comments. We can also see the user who raised an approval request which I can Approve. For some images, I can also Cancel Request, and that’s because I have raised an approval request for those images. The one who raised an approval request cannot approve the self-requests.

It is important to note that, with Devtron you get the entire audit trail of who approved the request, the user who raised a request as well as you can also see the user who has deployed it in the production environment in Deployment History. Additionally, you can also see the details of the configurations of the last deployment, compare them with the older releases and if anything breaks down, can easily rollback with a single click.

Step 4: With Devtron’s single pane of glass, you can also check out the deployment metrics critical for your business such as Deployment Frequency, Change Failure Rate, MTR, and MLT for your production environment as you can see in the below image.

Conclusion

In today’s dynamic software landscape, maintaining control over production deployments is more critical than ever. By implementing approval-based deployments for Kubernetes environments, organizations can enhance security, improve stability, ensure compliance, and boost operational efficiency. This practice not only protects the production environment but also fosters a culture of responsibility and continuous improvement within the team. With Devtron, it becomes much easier to set up guardrails around the Kubernetes ecosystem, natively integrating the gatekeeper within your workflows. As you embark on this journey, remember that the gatekeeper’s role is not to slow down progress but to safeguard the environment, ensuring that only the best, most secure code makes it to production.

If you have any questions, feel free to join our Community Discord Server and shoot out your questions, would be happy to answer them.