Understanding the Basics of a Canary Deployment Strategy

Introduction

Inspired by the use of canaries in British coal mines to detect toxic gases, canary deployment (aka. canary release) has become a crucial technique in software deployment. Just as canaries serve as early warning systems, canary deployment allows DevOps engineers to assess the impact of new software releases before introducing them to the entire user base. This comprehensive blog will explore the intricacies of canary deployment, including its process, benefits, considerations, and role in mitigating risks during the software release cycle.

What is Canary Deployment?

Canary Deployment is a release strategy designed to minimize the risks of introducing new software updates into production environments. It involves gradually rolling out changes to a small subset of users or infrastructure before making them available to a broader audience. By doing so, canary release enables early detection of any issues or performance degradation, allowing for prompt remediation and minimizing user impact. It’s important to note that you don’t need to consider deployment patterns during the continuous integration phase of your software development lifecycle.

When to Implement Canary Deployment?

Canary deployment is particularly beneficial when releasing new versions of software that include significant functionality updates or when there is a high risk of encountering issues in the production environment. Its primary objective is to catch and rectify problems before they affect the entire user base. Organizations can confidently enhance their software and services by adopting canary deployment, improving customer experience while mitigating potential risks.

Canary is a deployment pattern that is implemented by your continuous delivery or continuous deployment workflows. This pattern is considered to be a “zero downtime” deployment pattern since you do not stop the production environment at all during deployment or rollback.

Typical Canary Deployment Process:

The canary release process typically involves the following stages:

Plan and Create: The initial step revolves around creating a dedicated canary infrastructure where the latest update is deployed. This is not a new environment, rather it is an extension of the existing environment nodes where the new deployment will occur. A small portion of the overall traffic is directed to this canary instance, while most users continue to use the baseline version.

Analyze: Once the canary instance receives traffic, the team collects various data points, including metrics, logs, network traffic information, and results from synthetic transaction monitors. This data is then analyzed to determine whether the canary instance is operating as expected. A comparison with the baseline version helps identify any discrepancies or performance issues.

Rollout: After completing the canary analysis, the team decides whether to proceed with the release and roll it out to the rest of the users or back to the previous baseline state. The decision is based on the analysis results, ensuring the new release meets the required standards and doesn't adversely impact the user experience.

Benefits of Canary Deployment

Canary deployment offers several benefits that contribute to smoother software releases and reduced risks:

Zero Downtime with Faster Rollback: Canary deployment allows for quick rollbacks in case of issues or errors. By simply re-routing traffic back to the baseline instance, any problems are immediately eliminated from the customer's perspective. Engineers can then identify and resolve the root cause before introducing a new update, ensuring minimal disruption.

Cost-Efficiency with Small Infrastructure: Since canary release operates on a small subset of users or infrastructure, it requires only minimal additional resources. Unlike other deployment strategies like blue-green deployments, which necessitate provisioning entirely new infrastructure, canary deployment allows for changes to be tested on a smaller scale, reducing costs.

Flexibility for Innovation: By testing the canary instance with minimal impact on the overall user experience and infrastructure, developers gain the confidence to experiment and innovate. Canary deployment fosters a culture of continuous improvement, enabling organizations to introduce new features and enhancements while maintaining stability.

A/B Testing Capabilities: Canary release facilitates A/B testing, allowing organizations to compare the canary instance with the previous or old version on various metrics and assess its stability. Organizations can test their production stability and gather valuable insights by progressively increasing the load on the canary version before they migrate all instances.

Does Canary Deployment Work for All Deployment Sizes?

Canary deployment is well-suited for organizations of various sizes and deployment scenarios. Its inherent characteristics, such as short deployment cycles and the ability to target specific user subsets or infrastructure, make it conducive to fast and frequent updates. This agility benefits organizations by reducing time to market and providing customers with more value in less time. Additionally, canary release is particularly effective for large and distributed systems, offering flexibility to defer updates based on regional risk assessments in organizations with a global presence.

DevOps Role with Canary Deployment Strategy

Canary deployments can be tricky to implement without the right tooling to make them easier. That’s where the DevOps team comes into play. Platform Engineering, DevOps, Site Reliability Engineering (SRE), or similar teams with other names are typically responsible for providing the orchestration tooling that makes it possible for Development teams to implement the various deployment patterns such as rolling deployments, blue-green deployments, and canary deployments. With the right tooling in place, all of these advanced deployment patterns can be defined in templates and parameterized.

What about Feature Flags and Progressive Delivery?

Feature flags are NOT a delivery pattern. After you have deployed your new version of an application you can use feature flags to selectively enable and disable functionality as desired. Progressive delivery is a process of gradually turning on feature flags for portions of your user base (similar to a canary deployment strategy) and rolling back by disabling the feature flag if your observability indicates that the new functionality is not working properly. Feature flags are a secondary consideration beyond deployment strategy and are not in the scope of this article.

Limitations of Canary Deployments

While canary deployment provides significant advantages, it's essential to be mindful of specific considerations and limitations:

Automation for Efficiency: Conducting canary analysis manually can be time-consuming and error-prone, especially in complex deployment pipelines. To overcome this, organizations can leverage automation tools and platforms that streamline the analysis phase and improve the speed and reliability of the CI/CD process.

Challenges with On-Premises or Thick Client Applications: Implementing canary deployment can be challenging when applications are installed on personal devices. However, establishing an auto-update environment for end-users can help overcome this limitation and enable a smoother canary release process.

Complexities in Managing Databases: While managing different versions of the application with canary deployment is relatively smooth, handling database modifications can introduce complexity. Adapting the application to interact with multiple database instances and coordinating schema changes can require specialized skills and careful planning.

Canary vs Blue-Green Deployment

Canary deployment and Blue-Green deployment are two distinct approaches to releasing new software versions. Canary deployment takes a gradual and controlled approach, exposing a small subset of users or traffic to the latest release while most continue using the stable version. It's like a safety net that allows organizations to collect real-time feedback and closely monitor the new version's performance in a production-like environment. This way, any issues or unexpected behavior can be quickly identified and addressed before affecting a more extensive user base. Canary deployment adds an extra layer of validation, mitigating risks and ensuring a smooth transition to the new release.

On the other hand, Blue-Green Deployment focuses on maintaining two identical environments: the blue environment, representing the stable and currently active version, and the green environment, where the new version is deployed. The magic happens when the green environment is thoroughly tested and ready. Traffic is instantly redirected from the blue to the green environment, effectively switching user access to the new version. This approach ensures minimal downtime and provides a straightforward rollback mechanism by quickly redirecting traffic to the blue environment if any issues arise. With a strong emphasis on stability, reliability, and complete separation of environments, Blue-Green deployment is preferred for organizations prioritizing uninterrupted service and a consistent user experience throughout the deployment process.

In summary, while Canary deployment gradually exposes the new version to a subset of users for thorough testing, Blue-Green deployment maintains two separate environments to enable seamless user traffic switching between the stable and new versions. Both strategies offer unique benefits and cater to different deployment needs, allowing organizations to release software updates with minimal risks and disruptions to the end-user experience.

Influence of Kubernetes on Canary Deployment

The rise in the adoption of Kubernetes has sparked a surge in the popularity of the Canary Deployment strategy. Let us understand why.

Containers are now a standard infrastructure for hosting modern microservices architecture. They are lightweight and offer isolated environments, making it effortless to replicate application instances and manage different versions easily. Containers enabled rapid creation and deletion of infrastructure, allowing organizations to deploy new versions faster. While load balancers did a fine job of switching traffic seamlessly, it became difficult to manage at scale as the process was mostly manual. With Kubernetes in the picture, things changed.

With its ingress automation capabilities, Kubernetes automated the entire process, from deploying new versions to monitoring performance, collecting metrics, and intelligently routing traffic based on predefined rules. This automation streamlines the deployment process, making it more efficient and less error-prone.

Get Started with Canary Deployments using NGINX Ingress: How to Execute Canary Deployments Using NGINX Ingress (devtron.ai)

How To Do a Canary Deployment on Kubernetes - The Hard Way

Now that we know all about canary deployment strategies, let's take a look at how you go about implementing one.

Prerequisites

• A Kubernetes cluster
• kubectl installed and configured to communicate with your cluster
• An application deployed on Kubernetes

Step 1: Define Your Deployments

You need two deployments: one for the stable version of your application (primary) and one for the new version (canary). These deployments should be identical except for the version of the application they run and their labels.

Primary Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-primary
spec:
  replicas: 5
  selector: 
    matchLabels: 
      app: myapp 
      version: primary 
  template: 
    metadata: 
      labels: 
        app: myapp
        version: primary 
    spec: 
      containers: 
      - name: myapp 
        image: myapp:1.0

Canary Deployment:

apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: myapp-canary 
spec: 
  replicas: 1
  selector: matchLabels: 
    app: myapp 
    version: canary 
  template: 
    metadata: 
      labels: 
        app: myapp
        version: canary 
    spec: 
      containers: 
      - name: myapp 
        image: myapp:1.1

Step 2: Create a Service

Deploy a Kubernetes service that targets both the primary and the canary deployments. The service acts as the entry point for traffic to your application.

apiVersion: v1 
kind: Service 
metadata: 
  name: myapp-service 
spec: 
  ports: 
  - port: 80
    targetPort: 8080 
  selector: 
    app: myapp

Step 3: Route Traffic

To control the traffic flow to your primary and canary deployments, you can use Kubernetes itself or an Ingress controller with advanced routing capabilities (like Istio, Linkerd, or Traefik). For simplicity, we'll stick with Kubernetes' native capabilities.

Option A: Manual RoutingManually adjust the number of replicas in your deployments to control the traffic split. For example, to increase traffic to your canary, you would decrease replicas in the primary deployment and increase replicas in the canary deployment.

Option B: Automated Routing with Ingress ControllerUsing an Ingress controller that supports advanced routing rules allows you to specify weights for traffic distribution between your primary and canary deployments. This approach requires additional setup for the Ingress controller and defining routing rules.

Step 4: Monitor and Scale

Monitor the performance and stability of the canary deployment. Tools like Prometheus, Grafana, or Kubernetes' own metrics can help you gauge how the new version is performing compared to the stable version.

Step 5: Full Rollout or Rollback

Based on the performance and feedback:

• If successful, gradually shift more traffic to the canary version by adjusting the number of replicas until the canary deployment handles all traffic. Finally, update the primary deployment to the new version.
• If issues arise, rollback by shifting traffic away from the canary deployment back to the primary.

You'll need to repeat this process every time you want to do a new canary deployment. 😢

How To Do a Canary Deployment on Kubernetes - The Easy Way

Now that we've seen the difficult, manual method for rolling out a canary deployment on K8s, let's look at the process when you use Devtron.

Prerequisites

• Have a Kubernetes cluster with Istio service mesh installed.
• Install Flagger, which automates the promotion of canary deployments using Istio’s routing capabilities.
• Install Devtron for simplified CI/CD, monitoring, and observability on Kubernetes.

Read more about this automated canary deployment process.

Step 1: Configure Canary Deployment in Devtron

Deploy Your Application: Use Devtron’s user-friendly dashboard to deploy your application onto the Kubernetes cluster. This involves setting up the application’s Docker image, configuring resources, and defining environment variables.
Set Up Canary Analysis: Configure canary analysis strategies in Devtron, specifying metrics that Flagger should monitor during the canary deployment. These metrics could include success rates, request durations, and error rates, ensuring that the new version meets your defined criteria for stability and performance.

Step 2: Automate Rollout with Flagger

Define Canary Custom Resource: Create a Canary custom resource definition (CRD) in Kubernetes that specifies the target deployment, the desired canary analysis strategy, and rollback thresholds. This CRD instructs Flagger on how to manage the canary deployment process.

Monitor Deployment Progress: Flagger, integrated with Prometheus, automatically monitors the defined metrics during the canary rollout. If the new version underperforms or fails to meet the criteria, Flagger halts the rollout and triggers a rollback to the stable version.

Step 3: Leverage Istio's Traffic Management

Control Traffic Split: Flagger uses Istio to dynamically adjust the traffic split between the stable and canary versions during the deployment, starting with a small percentage of traffic to the canary and gradually increasing it as the canary proves stable. Check out this blog for canary deployments with Flagger and Istio.

Step 4: Observability and Monitoring

Devtron provides integrated monitoring and observability features, offering insights into the deployment process, application performance, and user impact. Utilize these tools to closely monitor the canary deployment and make informed decisions.

Step 5: Finalize Deployment

Upon successful canary analysis, Flagger gradually shifts all traffic to the new version. If the canary meets all criteria, it's promoted to a stable release, and the old version is phased out. If issues arise, Flagger automatically rolls back to the stable version, minimizing the impact on users.

This automated canary deployment process will happen every time you trigger a new deployment from Devtron. There's no ongoing work required. 😀

Conclusion

Canary deployment has emerged as a valuable strategy for minimizing risks and ensuring smooth software releases. By gradually rolling out updates and monitoring the performance of a canary instance, organizations can proactively address any issues before they impact the entire user base. With benefits such as zero production downtime, cost-efficiency, flexibility for innovation, and A/B testing capabilities, canary deployment empowers organizations to balance continuous improvement and user satisfaction.

To fully leverage Canary deployment, organizations can adopt Platforms such as Devtron that provide such strategies out of the box. By doing so you will realize the benefits of zero downtime testing of new versions of the application, easy rollbacks, and little to no API or scripting work.