How to Manage Secrets with Azure Key Vault in Kubernetes?

Securing sensitive data and information has become paramount in today's world. The Azure Key Vault offers a robust solution for centrally managing and protecting secrets like passwords, API keys, and certificates. By integrating Azure Key Vault with Kubernetes, we can streamline the process of managing sensitive information, enhancing security, and reducing the risk of data breaches. In this blog post, we'll delve into the steps involved in leveraging Azure Key Vault within a Kubernetes environment, to ensure the security of applications and how we can securely access and utilize secrets in Kubernetes.

What is Azure Key Vault?

Azure Key Vault is a cloud service provided by Microsoft Azure that allows you to securely manage and store sensitive information such as passwords, authentication tokens, encryption keys, and certificates. It offers centralized control over access to these secrets, helping you safeguard your application's sensitive data. Key vaults can also help you control and manage access to keys and secrets, which are critical components for securing data in the cloud.

Components of Key Vaults

It is composed of three primary components:

1. Keys: Azure Key Vault allows you to store and manage cryptographic keys. These keys can be used to encrypt and decrypt data, and also to authenticate users and services.
2. Secrets: Azure Key Vault also provides secure storage for sensitive information, such as passwords, connection strings, and API keys. You can use these secrets in your applications without exposing them directly.
3. Certificates: Azure Key Vault allows you to manage and store certificates used for authentication and encryption.

In this blog, we will delve into the intricacies of managing secrets stored in Azure Key Vaults within the context of a Kubernetes environment.

Create Azure Key Vault & Store Token in Vault

Now, let’s create an Azure key vault and store a token in it, which would be fetched in Kubernetes. Let's create a service principal assign the necessary permissions for Key Vaults and authenticate the user using the “az login” command in your terminal.

Execute the following commands in sequence to create a key vault and store secrets in it.

Create a resource group in the central India region

az group create -l centralindia -n automation-rg

Create a Key Vault named "backend-app-keyvault" in the previously created resource group and the central India region:

az keyvault create -n backend-app-keyvault -g automation-rg -l centralindia

Set a secret within the Key Vault. Use the following command to set a secret named "backend-api-token" with the value "Backend09872" in the "backend-app-keyvault"

az keyvault secret set --vault-name backend-app-keyvault -n backend-api-token --value Backend09872

Now we have successfully created a vault and stored a secret. Let’s move further and pull that secret in the Kubernetes cluster. For application deployments on Kubernetes and integrating with Azure Key Vault, we would be using Devtron. Please look at the prerequisites and ensure it has been completed.

Deploying Azure Key Vault Secrets on Kubernetes using Devtron

Now, let's move ahead and integrate the Azure Key Vault Secrets with Devtron, and deploy it on Kubernetes.

Prerequisites

Devtron Installation: Confirm that Devtron is installed on your Kubernetes cluster. We would be using Devtron for all our deployments and integration with Azure Key Vault. It can be installed with the following commands,

helm repo add devtron https://helm.devtron.ai 

helm repo update devtron

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd}

External Secrets Operator: Install the ESO (External Secrets Operator) using Helm Chart. ESO enables seamless integration between Kubernetes and external secret management systems like Azure Key Vault and it has been natively integrated within Devtron.

Helm chart link: https://charts.external-secrets.io

Once ESO has been deployed, let’s verify that we have successfully installed the ESO operator. This operator facilitates the synchronization of secrets from external sources, such as Azure Key Vault, to Kubernetes secrets.

Next, it's essential to create a service principal to establish authentication with Azure Key Vault. This authentication mechanism will enable secure communication between your Kubernetes environment and the Key Vault.

Here's a detailed overview of the steps involved:

• Create Service Principal: Begin by generating a service principal in Azure. This principle will act as a secure identity for accessing Azure resources, including the Key Vault.
• Assign Permissions: Assign the necessary permissions to the service principal. These permissions will determine the level of access the principal has to interact with Azure Key Vault.

By creating a service principal and enabling proper authentication, you establish a secure connection between your Kubernetes environment and Azure Key Vault. This is a crucial foundation for managing and utilizing secrets effectively within your applications.

Now, let's proceed with creating an application using Devtron, where we'll integrate the previously mentioned secret data. Follow along the steps to configure & deploy applications using Devtron.

Step 1:  Setup your Git Repository

Step 2: Provide the Build Configuration of the app as shown below screenshot

Step 3: Generating Kubernetes Secrets

Generate a Kubernetes secret containing the credentials of the service principal created in the previous step. These credentials will be used to authenticate with Azure Key Vaults. Refer to the screenshot below for  guidance

Step 4: Deploy the Application through Devtron

Now, let's proceed with deploying our application to the Kubernetes cluster. For a detailed guide on how to deploy applications to Kubernetes using Devtron, you can refer to our blog. Upon deploying this application, you will find that a Kubernetes secret named azure-auth-service-principal-1713 has been generated. This secret holds the essential credentials of the service principal. These credentials play a crucial role in facilitating the functioning of the SecretStore Kubernetes object. We can see this in the screenshot below.

Authentication for Azure Key Vaults has been successfully completed. Next, proceed to select the option for creating an external secret within Azure Secret Manager. In this step, you will need to provide essential information about your Key Vaults.

Step-5: Authentication in SecretStore

For authenticating with Azure Key Vaults within the SecretStore, the authSecretRef is utilized. Additionally, you will need to include your tenant ID and the vault URL in the Azure Key Vault section as mentioned below screenshot.

Upon initiating the deployment with the aforementioned input data, the backend-API-token is now exposed as an environment variable within the application pod. Here, we can observe that the application is in a healthy state, and both the SecretStore and the External Secret have been successfully created.

Now Access the pod terminal and examine the value of the backend-api-token.

Now, let's modify the value of the API token in Azure Key Vaults UI. Afterward, the ESO controller triggers the reconciliation of the ExternalSecret. This will update the value of the API token in the secret that was provisioned by the ExternalSecret.

The updated value will be reflected in the pod after the refresh interval specified in the previous step as we are seeing in the environment variable in the pod.

Conclusion

In this blog, we have successfully integrated Azure Key Vault with Kubernetes. We have discussed some key components of Azure Key Vault like Keys, Secrets, and Certificates. Moreover, we have also seen how one can deploy Azure Key Vault on Kubernetes using Devtron and ESO. By integrating Azure Key Vault with Kubernetes using Devtron and the External Secrets Operator (ESO), we can establish a robust and secure way to manage secrets for our applications. This approach allows you to centrally store and manage sensitive data while ensuring proper access control.

If you have any questions or concerns, please join our dedicated Discord community server and don't hesitate to reach out to us.