Understanding the Basics of a Canary Deployment Strategy

Introduction

Inspired by the use of canaries in British coal mines to detect toxic gases, canary deployment (aka canary release) has become a crucial technique in software deployment. Just as canaries serve as early warning systems, canary deployment allows DevOps engineers to assess the impact of new software releases before introducing them to the entire user base. This comprehensive blog will delve into the intricacies of canary deployment, including its process, benefits, key considerations, and its role in mitigating risks during the software release cycle.

When to Implement Canary Deployment?

Canary deployment is particularly beneficial when releasing new versions of software that include significant functionality updates or when there is a high risk of encountering issues in the production environment. Its primary objective is to catch and rectify problems before they affect the entire user base. Organizations can confidently enhance their software and services by adopting canary deployment, improving customer experience while mitigating potential risks.

Canary is a deployment pattern that is implemented by your continuous delivery or continuous deployment workflows. This pattern is considered to be a “zero downtime” deployment pattern since you do not stop the production environment at all during deployment or rollback.

Typical Canary Deployment Process:

The canary release process typically involves the following stages:

Plan and Create: The initial step revolves around creating a dedicated canary infrastructure where the latest update is deployed. This is not a new environment, rather, it is an extension of the existing environment nodes where the new deployment will occur. A small portion of the overall traffic is directed to this canary instance, while most users continue to use the baseline version.

Analyze: Once the canary instance receives traffic, the team collects various data points, including metrics, logs, network traffic information, and results from synthetic transaction monitors. This data is then analyzed to determine whether the canary instance is operating as expected. A comparison with the baseline version helps identify any discrepancies or performance issues.

Rollout: After completing the canary analysis, the team decides whether to proceed with the release and roll it out to the rest of the users or back to the previous baseline state. The decision is based on the analysis results, ensuring the new release meets the required standards and doesn't adversely impact the user experience.

Does Canary Deployment Work for All Deployment Sizes?

Canary deployment is well-suited for organizations of various sizes and deployment scenarios. Its inherent characteristics, such as short deployment cycles and the ability to target specific user subsets or infrastructure, make it conducive to fast and frequent updates. This agility benefits organizations by reducing time to market and providing customers with more value in less time. Additionally, canary release is particularly effective for large and distributed systems, offering flexibility to defer updates based on regional risk assessments in organizations with a global presence.

DevOps Role with Canary Deployment Strategy

Canary deployments can be tricky to implement without the right tooling to make them easier. That’s where the DevOps team comes into play. Platform Engineering, DevOps, Site Reliability Engineering (SRE), or similar teams with other names are typically responsible for providing the orchestration tooling that makes it possible for Development teams to implement the various deployment patterns such as rolling deployments, blue-green deployments, and canary deployments. With the right tooling in place, all of these advanced deployment patterns can be defined in templates and parameterized.

What about Feature Flags and Progressive Delivery?

Feature flags are NOT a delivery pattern. After you have deployed your new version of an application, you can use feature flags to selectively enable and disable functionality as desired. Progressive delivery is a process of gradually turning on feature flags for portions of your user base (similar to a canary deployment strategy) and rolling back by disabling the feature flag if your observability indicates that the new functionality is not working properly. Feature flags are a secondary consideration beyond deployment strategy and are not in the scope of this article.

Influence of Kubernetes on Canary Deployment

The rise in the adoption of Kubernetes has sparked a surge in the popularity of the Canary Deployment strategy. Let us understand why.

Containers are now a standard infrastructure for hosting modern microservices architecture. They are lightweight and offer isolated environments, making it effortless to replicate application instances and manage different versions easily. Containers enabled rapid creation and deletion of infrastructure, allowing organizations to deploy new versions faster. While load balancers did a fine job of switching traffic seamlessly, it became difficult to manage at scale as the process was mostly manual. With Kubernetes in the picture, things changed.

With its ingress automation capabilities, Kubernetes automated the entire process, from deploying new versions to monitoring performance, collecting metrics, and intelligently routing traffic based on predefined rules. This automation streamlines the deployment process, making it more efficient and less error-prone.

Get Started with Canary Deployments using NGINX Ingress: How to Execute Canary Deployments Using NGINX Ingress (devtron.ai)

How To Do a Canary Deployment on Kubernetes - The Hard Way

Now that we know all about canary deployment strategies, let's take a look at how you go about implementing one.

Prerequisites

• A Kubernetes cluster
• kubectl installed and configured to communicate with your cluster
• An application deployed on Kubernetes

Step 1: Define Your Deployments

You need two deployments: one for the stable version of your application (primary) and one for the new version (canary). These deployments should be identical except for the version of the application they run and their labels.

Primary Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-primary
spec:
  replicas: 5
  selector: 
    matchLabels: 
      app: myapp 
      version: primary 
  template: 
    metadata: 
      labels: 
        app: myapp
        version: primary 
    spec: 
      containers: 
      - name: myapp 
        image: myapp:1.0

Canary Deployment:

apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: myapp-canary 
spec: 
  replicas: 1
  selector: matchLabels: 
    app: myapp 
    version: canary 
  template: 
    metadata: 
      labels: 
        app: myapp
        version: canary 
    spec: 
      containers: 
      - name: myapp 
        image: myapp:1.1

Step 2: Create a Service

Deploy a Kubernetes service that targets both the primary and the canary deployments. The service acts as the entry point for traffic to your application.

apiVersion: v1 
kind: Service 
metadata: 
  name: myapp-service 
spec: 
  ports: 
  - port: 80
    targetPort: 8080 
  selector: 
    app: myapp

Step 3: Route Traffic

To control the traffic flow to your primary and canary deployments, you can use Kubernetes itself or an Ingress controller with advanced routing capabilities (like Istio, Linkerd, or Traefik). For simplicity, we'll stick with Kubernetes' native capabilities.

Option A: Manual RoutingManually adjust the number of replicas in your deployments to control the traffic split. For example, to increase traffic to your canary, you would decrease replicas in the primary deployment and increase replicas in the canary deployment.

Option B: Automated Routing with Ingress ControllerUsing an Ingress controller that supports advanced routing rules allows you to specify weights for traffic distribution between your primary and canary deployments. This approach requires additional setup for the Ingress controller and defining routing rules.

Step 4: Monitor and Scale

Monitor the performance and stability of the canary deployment. Tools like Prometheus, Grafana, or Kubernetes' own metrics can help you gauge how the new version is performing compared to the stable version.

Step 5: Full Rollout or Rollback

Based on the performance and feedback:

• If successful, gradually shift more traffic to the canary version by adjusting the number of replicas until the canary deployment handles all traffic. Finally, update the primary deployment to the new version.
• If issues arise, rollback by shifting traffic away from the canary deployment back to the primary.

You'll need to repeat this process every time you want to do a new canary deployment. 😢

How To Do a Canary Deployment on Kubernetes - The Easy Way

Now that we've seen the difficult, manual method for rolling out a canary deployment on K8s, let's look at the process when you use Devtron.

Prerequisites

• Have a Kubernetes cluster with Istio service mesh installed.
• Install Flagger, which automates the promotion of canary deployments using Istio’s routing capabilities.
• Install Devtron for simplified CI/CD, monitoring, and observability on Kubernetes.

Read more about this automated canary deployment process.

Step 1: Configure Canary Deployment in Devtron

Deploy Your Application: Use Devtron’s user-friendly dashboard to deploy your application onto the Kubernetes cluster. This involves setting up the application’s Docker image, configuring resources, and defining environment variables.
Set Up Canary Analysis: Configure canary analysis strategies in Devtron, specifying metrics that Flagger should monitor during the canary deployment. These metrics could include success rates, request durations, and error rates, ensuring that the new version meets your defined criteria for stability and performance.

Step 2: Automate Rollout with Flagger

Define Canary Custom Resource: Create a Canary custom resource definition (CRD) in Kubernetes that specifies the target deployment, the desired canary analysis strategy, and rollback thresholds. This CRD instructs Flagger on how to manage the canary deployment process.

Monitor Deployment Progress: Flagger, integrated with Prometheus, automatically monitors the defined metrics during the canary rollout. If the new version underperforms or fails to meet the criteria, Flagger halts the rollout and triggers a rollback to the stable version.

Step 3: Leverage Istio's Traffic Management

Control Traffic Split: Flagger uses Istio to dynamically adjust the traffic split between the stable and canary versions during the deployment, starting with a small percentage of traffic to the canary and gradually increasing it as the canary proves stable. Check out this blog for canary deployments with Flagger and Istio.

Step 4: Observability and Monitoring

Devtron provides integrated monitoring and observability features, offering insights into the deployment process, application performance, and user impact. Utilize these tools to closely monitor the canary deployment and make informed decisions.

Step 5: Finalize Deployment

Upon successful canary analysis, Flagger gradually shifts all traffic to the new version. If the canary meets all criteria, it's promoted to a stable release, and the old version is phased out. If issues arise, Flagger automatically rolls back to the stable version, minimizing the impact on users.

This automated canary deployment process will happen every time you trigger a new deployment from Devtron. There's no ongoing work required. 😀

Conclusion

Canary deployment has emerged as a valuable strategy for minimizing risks and ensuring smooth software releases. By gradually rolling out updates and monitoring the performance of a canary instance, organizations can proactively address any issues before they impact the entire user base. With benefits such as zero production downtime, cost-efficiency, flexibility for innovation, and A/B testing capabilities, canary deployment empowers organizations to balance continuous improvement and user satisfaction.

To fully leverage Canary deployment, organizations can adopt Platforms such as Devtron that provide such strategies out of the box. By doing so you will realize the benefits of zero downtime testing of new versions of the application, easy rollbacks, and little to no API or scripting work.

FAQ

Prakarsh

Prakarsh is Head of DevOps at Devtron Inc, with 15+ years in the DevOps ecosystem. He drives automation, cloud adoption & security, optimizing processes for scalable, production-ready solutions.

Jyoti Sahoo

Jyoti crafts engaging content for business and tech leaders. He has led AI projects for EU governments and ANZ clients. A 3D printing enthusiast and solar energy advocate.

Co-authors: Prakarsh, Jyoti Sahoo,