There has been a paradigm shift in the software industry where organizations are adopting microservice architecture to achieve scale, flexibility, and high availability of services. At the same time, when this paradigm helps organizations to scale and operate smoothly, configuration management in microservice architecture presents a significant challenge. As applications and services evolve to achieve scale, the number of configuration variables expands exponentially. This expansion of the variables creates a complex web, where each one needs to be handled carefully as they carry crucial configurations and secrets for the application.
The complex web of multiple configuration variables slows down teams; the development team needs to keep track of multiple environment-specific variables, while the operation team struggles to manage consistency across multiple environments. With all these challenges, the objective of the organization to achieve scale and operate at lightning speed appears to be fading. The complexities start to multiply with Kubernetes, where the applications and services are spread across multiple pods, namespaces, and Kubernetes clusters.
A centralized configuration management system will be beneficial for simplifying variable management across multiple microservices and environments. In this blog, we will discuss the challenges faced while managing Kubernetes configurations at scale, also we will take a look at a centralized system for managing configuration in Kubernetes to help us scale and navigate the complexities of configuration management.
We will look at real-world scenarios where organizations managing hundreds or even thousands of microservices have successfully implemented centralized configurations for Kubernetes resources and secret management through scoped variables and achieved operational efficiency.
Variable Overload: The Scaling Hurdle
Managing Kubernetes configurations at scale for 100s of microservices is still an overhead. For instance, consider that you are at an organization that has adopted Kubernetes to leverage the power of Kubernetes to scale and get flexibility. Your teams started deploying multiple services across multiple nodes and even across multiple Kubernetes clusters at lightning speed. At a certain scale, your teams will start facing issues like
Inconsistency: As your microservices scale across environments, maintaining consistency across configurations becomes difficult for the operations teams. Without having a centralized way, teams need to manage all these variables manually, which is quite a tedious and error-prone method. As these variables carry some crucial configurations in them, a small mistake can cause inconsistency or even downtime for service.
Collaboration: As your organization scales its Kubernetes infrastructure, multiple teams working on the same applications struggle to coordinate effectively. The team can face issues like
- Sharing exact configurations between teams manually may increase the chances of misconfigurations and disruptions.
- Teams overwrite each other's configurations, leading to conflicts and the unexpected working of applications.
- Having no clear accountability or control over configurations getting changed.
Security: The critical data like API keys, database credentials, and other secrets that have been stored in these configuration variables for the ease of accessibility or rotation of credentials will remain scattered throughout the environments. Increasing the surface area for the attackers.
Version Control: As your applications will evolve so their configurations will. Keeping track of changing configurations of these each release turns out to be a nightmare for teams. Your teams might struggle to maintain a history of configuration changes, making it difficult to troubleshoot if there are any issues. Additionally, you can maintain the configurations at version control systems like GitHub, but managing the configurations and changing the configurations for different environments can be troublesome.
Scalability: The challenges that we mentioned above are the ones that are most likely to be faced without having a centralized configuration system. While Kubernetes excels at scaling and providing flexibility, the practices that you followed to manage the configuration will be not able to keep pace with Kubernetes and teams.
The complex web of configurations, scattered across environments creates the challenges mentioned above and hinders the scalability of services.
The Missing Piece: Centralized Management for Kubernetes Configurations
To address the missing piece, Devtron has recently launched a feature called Scoped Variable, which has been used by customers and open-source users actively. The Scoped Variable of Devtron allows you to declare variables and their values at a centralized location. These configuration variables can be accessed across the Kubernetes configurations, pods, namespaces, and clusters. Like all other Kubernetes resources, Devtron allows us to define our configuration variables in a declarative way, i.e., in JSON or YAML format. Devtron provides a template where we can define our configuration variables, define the scope of these variables, and upload them to the Devtron dashboard. Once done, the defined variables will be available for applications and environments based on the scope we defined for them.
Sample template to define the Configurations and their Scope across environments
apiVersion: devtron.ai/v1beta1
kind: Variable
spec:
# Name of the variable, must be unique
- name: KAFKA
#Provide a short description for this variable in max 120 characters. This description will be shown in UI while using the variable.
shortDescription: Enter a short description here
# Mention notes that tell more about this variable. This will not be shown in UI while using the variable. No char limit.
notes: Enter any notes for additional details
# Optional property. Accepts Boolean Value, default value false. Values for sensitive variables are not shown in UI while using the variable
isSensitive: false
# List of values for this variable
values:
#Global category can be used to define variables which are common across all applications
- category: Global
value: Global
#Cluster category can be used to define variables which are common across all environments of a particular cluster
- category: Cluster
value: Cluster-8080
selectors:
attributeSelectors:
# ClusterName is required for cluster category
ClusterName: prod-cluster
#Application category can be used to define variables which are common across all workflows of a particular applications
- category: Application
value: Application-8080
selectors:
attributeSelectors:
# ApplicationName is required for application category
ApplicationName: dev-test-1
#Env category can be used to define variables which are common across all applications deployed on a particular environment
- category: Env
value: Prod-Env
selectors:
attributeSelectors:
#EnvName is required for env category
EnvName: dev-test-1
#ApplicationEnv category can be used to define variables which are scoped for a particular environment of an application
- category: ApplicationEnv
value: Application-8080
selectors:
attributeSelectors:
# ApplicationName is required for applicationEnv category
ApplicationName: dev-test
# EnvName is required for applicationEnv category
EnvName: virtual-env-1
Why Scoped Variable?
When it comes to configuration management at scale for Kubernetes workloads, the Scoped Variable plays a vital role. With its capability to define variables and their scope at different levels, such as Global - true for multi-clusters, Environment-specific, Application specific, it makes it much easier to manage configurations seamlessly for different environments and multi-clusters. Additionally, these variables can also be used by Helm charts that you deploy through Devtron along with pre/post build stages and pre/post deployments stages, including the plugins. Some of the advantages of using Scoped Variable are:
Centralized Management: As with the Scoped Variables, we define all the configurations in a single place and are able to use them across the scope defined, which acts as centralized configuration storage for all our environments. Even if we have 10 Kubernetes clusters up and running, some of them as productions and some as non-productions, in every cluster with multiple namespaces, pods, and applications, the scoped variables help us manage the complex web of configurations.
Improved Collaboration: As all the configurations are maintained in a single place, multiple teams can collaborate efficiently. The dependency on getting configuration from another team gets eliminated. For addressing the security concerns for the critical configurations, Devtron provides a fine-grained Resource Base Access Control (RBAC).
Version-Specific Configuration: In the Kubernetes environments, where you need to manage multiple services and their versions, each version comes with different and changed configurations. The Scoped Variables help you manage the versioned configurations. Also, in the case of updating the value of configurations, we can append at a single place, and it will be applied across environments. Eliminating the tedious process of manually going to each service and updating the configurations.
Traceability and auditing: Devtron provides us with the ability to trace the unwanted changes that happened in the configurations. The user who updated the configurations and what values have been updated can be traced by looking at the Devtron dashboard with a config diff view.
Reusability: Scoped Variables give us the accessibility to use the configurations at multiple levels of our environments. If we want to use the same configurations, we can define them as scoped variables and define the proper scope. The scoped variables act as the plug-in and plug-out configurations.
Enhanced Security: The centralized system improves security, and Devtron's RBAC helps manage proper access control on critical configurations.
Scalability: The Scoped Variables by Devtron enable the efficient management of configurations as your Kubernetes footprint expands. Declaring the configuration with appropriate scope allows simplified configuration management across multiple clusters, namespaces, and applications. The centralized approach allows you to add more environments and services without navigating the complexities of configuration management, as they are already stored in a centralized place. As infrastructure grows, scoped variables help maintain consistency and reduce the complexity of managing configurations across an expanding system.
Scoped Variable in Action
Now. let’s take a look at how to set our Centralized Configuration System through Scoped Variables
Step-1: Declare the variables Global Configuration > Scoped Variables > Upload file. Here we need to upload the JSON or YAML file, where we will declare the variables, their values, and the scope.
Let’s see an example of how to declare our variables and define their scope.
To learn how to define Scoped Variables please refer to the documentation. Once we upload the file, describing our configurations and variables we can visualize our variables in the variables tab.
Step 2: Accessing the configuration across environments and applications. The configuration can be accessed at the Deployment template where we need to describe the configurations before deployment.
By clicking on the Scoped Variables we get access to configuration variables. We will be using a scoped variable named HOST which holds the value for the ingress host domain.
In the case of integrating a plugin in the CI/CD pipeline, we can use these variables to store critical configurations like credentials or endpoints, for eg. we are using a scoped variable to store the SonarqubeApiKey.
As you can see above, we were able to centralize our Kubernetes configurations with Scoped Variables in just a few steps. We can reuse these Scoped Variables across Devtron at multiple stages. Now that we have a centralized place to store configurations and access them according to the defined scope, we can scale our Kubernetes infrastructure and services without going through tedious management methods.
Sailing the Seas of K8s Complexities with Pro Sailor: Devtron
Let’s look at some real-world examples where some big enterprises with thousands of microservices and multiple Kubernetes clusters are managing their Kubernetes world and its configurations with Devtron.
Navigating the Kubernetes Ocean: Tales of 4000 Microservices
An organization with a fleet of 4000 microservices and multiple Kubernetes clusters has been navigating the sea of complexities by integrating Devtron as a trusted navigator. For the last 4 years, they have been leveraging the power of devtron features like Kubernetes Management Dashboard, Policy Driven CI/CD Pipelines, Visibility for Helm Applications, Security Scannings, and precise RBAC policies for controlled user access.
In this blog, we'll focus on how they leverage the Scoped Variable feature to manage configurations for their 4000 microservices and look at some major use cases.
The Rotating Credentials at Scale
The organization has a policy of rotating the credentials for services and databases for security reasons. Imagine rotating the credentials of 4000 microservices manually, which is impractical, time-consuming, and also an error-prone method, a single mistake can disrupt services across the globe.
To streamline the process and avoid security issues, they have been storing the configurations and credentials in Devtron's Scoped Variables. When it's time to rotate credentials, they simply update the values of these variables, and the changes get applied across the environment and the scope of that variable.
Cross-Team Collaboration and Configuration Management
With the huge number of 4000 microservices running 24/7, multiple teams from different domains need to collaborate and share configurations and credentials. Devtron's Scoped Variables facilitate this collaboration by providing a centralized location for managing multiple configurations, protected by Devtron's fine-grained RBAC policies.
Streamlining Multi-Application Management at Scale
An Enterprise with an Umbrella Helm chart utilizes devtron to manage multiple applications on their Kubernetes cluster. As they have an Umbrella Helm chart by integrating Devtron they get visibility for all Helm applications through the Helm dashboard of Devtron.
Centralization of Configurations
Devtron’s Scoped Variables plays a crucial role in their setup of the Umbrella Helm chart. As they have multiple applications running across various environments, managing the configurations of these applications they require a centralized system. The place where they can store and use configurations anytime and anywhere. Scoped Variables provided the ideal solution, offering a single point of control for configurations that can be accessed across various environments. This approach significantly simplifies configuration management and enhances overall operational efficiency.
Ensuring Consistency Across Environments
The same enterprise faced another challenge in maintaining consistency across the sidecar containers. Many of their applications require sidecar containers, for metrics collection, and the same image must be used across all environments and pods.
To address this, the enterprise is leveraging Devtron's Scoped Variables feature. Where they store the sidecar container image address and the tag of the image as a variable and utilize the same variable whenever needed. This ensures uniformity across their entire infrastructure also the method guarantees that every instance of the sidecar container, regardless of the environment or pod it's deployed in, uses the same image.
There are multiple such stories for Scoped Variables, these were some major use cases of organizations using Devtron’s Scoped Variables as a Centralized Configuration Management system. Similarly, multiple large organizations are currently using Devtron to navigate the complexities of Kubernetes, making Devtron the captain of their ship. We'll be bringing you more interesting real-world stories of Devtron in action, so stay tuned!
Conclusion
Centralized configuration management is essential for organizations adopting microservices and Kubernetes. Without centralized management, multiple configurations and secrets form a complex web which hinders the scale and also increases the surface area for attackers. As a solution Devtron's Scoped Variables offer a centralized place to manage configurations across multiple environments, improving consistency and collaboration. The Real-world examples that we have seen above demonstrate how large enterprises successfully use Scoped Variables to manage configurations of thousands of microservices efficiently. The key benefits of Scoped Variables include centralized management of all configurations and secrets across multiple Kubernetes clusters, enhanced security, improved collaboration, traceability and auditing of changes, version control, and scalability.
