Setting up Monitoring Stack on Kubernetes with Devtron

Metrics, monitoring, and alerting are the key components of a monitoring system. It is vital to have a proper monitoring system enabled on Kubernetes clusters, to monitor and observe the deployed applications, as well as the Kubernetes infrastructure

How do you monitor your application? Chances are, if you're running a containerized application, then you are using Prometheus along with some other visualization/ dashboarding tool like Grafana, New-relic, or others. Prometheus is the defacto standard when it comes to monitoring your application.

Prometheus

Prometheus has been one of the go-to choices of cloud-native users for setting up monitoring stack. It is an open-source monitoring framework that comes with a time-series database, flexible query language, and a modern alerting mechanism. With Devtron's charts marketplace, you get all the community-trusted helm charts; if there aren't any, you can easily add them. Check out this blog for detailed insights about deploying helm charts with devtron.

Further in the blog, we will deploy a Prometheus helm chart and see how it can be used to get real-time metrics of all applications deployed through Devtron.

Grafana

Grafana is an open-source observability platform that helps you observe the metrics of your application with a beautiful dashboard and customization as per your needs. It is packaged in Devtron by default. To use the Grafana dashboard, you need to first install the integration from the Devtron Stack Manager. The beauty of integrations that Devtron brings in is that users don't have to worry about the different tools and their complexities. All the heavy lifting is carried out by Devtron and users just need to interact with the Devtron dashboard. In case, you want to access the Grafana dashboard, you need to port forward the devtron-grafana service if Devtron is running in the local system or exposes it via ingress.

kubectl -n devtroncd port-forward svc/devtron-grafana 3000:80 &

Go to your browser and type the URL, In our case, it is localhost:3000, and you should see this. Click on the sign-in button at the bottom left corner.

You should see this on the login page of Grafana.

To login into the Grafana dashboard, your username is admin and you can get your password by executing the following command:

kubectl get secrets -ndevtroncd devtron-grafana-cred-secret -ojsonpath='{.data.admin-password}'|base64 -d

Monitoring Stack with Devtron

Devtron has been built in a modular fashion where you can install different integrations as per your requirements. For monitoring, the default integration is Grafana, as mentioned above. Once Grafana has been installed, now we need to set up a metrics collector, which would scrape all the metrics and act as a data source for Grafana. In this tutorial, we will set up Prometheus and add it to the Devtron dashboard for fetching application metrics.  

Installing the required CRDs

Before we can go ahead and install Prometheus from the chart store, we first need to install the Custom Resource Definitions that are needed for running Prometheus. These CRDs do not get applied by the helm chart, hence we have to do it manually.

The manifest files that we will require for applying CRDs exist within the kube-prometheus GitHub repository. There are multiple manifest files that have to be applied. So let’s first go ahead and clone the repository. Run the below command to clone the repository to your local system.

git clone https://github.com/prometheus-operator/kube-prometheus && cd kube-prometheus

The manifests that we require exist within the manifests/setup directory. However, simply applying these manifests will throw an error. These manifests have to be applied on the server side. We can do this by passing the --server-side flag within the command. Run the below command to apply the CRDs.

kubectl apply --server-side -f manifests/setup

Alternatively, we can apply the CRDs directly from the raw files which exist within the same repository. If you wish to do that, run the below command:

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/0alertmanagerConfigCustomResourceDefinition.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/0alertmanagerCustomResourceDefinition.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/0podmonitorCustomResourceDefinition.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/0probeCustomResourceDefinition.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/0prometheusCustomResourceDefinition.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/0prometheusagentCustomResourceDefinition.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/0prometheusruleCustomResourceDefinition.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/0prometheusruleCustomResourceDefinition.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/0scrapeconfigCustomResourceDefinition.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/0servicemonitorCustomResourceDefinition.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/0thanosrulerCustomResourceDefinition.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup/namespace.yaml

Now that we have the CRDs applied within the cluster, we can go ahead and install Prometheus

Note: Please make sure that you apply the CRDs before installing the kube-promethus stack. Otherwise, certain components may not be created.

Installing Prometheus

Go to the chart store and search for Prometheus. Use the Prometheus-community's kube-prometheus-stack chart to deploy Prometheus. To learn more about this chart, check out the official chart page.

Once you select a chart, you must configure the values per your requirements before deploying. In our case, let's make the following changes.

kube-state-metrics: 
	metricLabelsAllowlist:   
	- pods=[*]
    
serviceMonitorSelectorNilUsesHelmValues: false

podMonitorSelectorNilUsesHelmValues: false

Search for the above parameters and set these values as defined above. Additionally, if you want to expose the Prometheus dashboard via ingress, you can enable the ingress and provide the hostname for it, and likewise for Grafana which is being bundled with kube-prometheus-chart

Now, after installing Prometheus, you need to get the endpoint of the Prometheus server. For every helm chart or application deployed at Devtron, you get a resource-grouped view for all Kubernetes resources deployed along with him. To get the endpoint, under Networking go-to service expand the Prometheus server service and you should be able to see the Endpoints along with EndpointSlice as shown below.

Enable Application Metrics

Not to use Prometheus as a data source for Grafana, go to Global Configurations -> Clusters & Environments and for the respective cluster where you have installed the Prometheus chart, add the endpoint as mentioned below and click on Update Cluster.

Once you add the endpoint, you will be able to see the application metrics in the Devtron dashboard for all the applications deployed in the respective cluster irrespective of the environment (ns) it is being deployed.

With this, you will be able to track your application metrics like resource usage, CPU usage, Throughput, and Latency. For Throughput and Latency, you need to enable the application metrics for respective environments from Deployment Template as mentioned below.

Conclusion

This is how you can set up a monitoring stack for Kubernetes with Devtron and easily observe and monitor all your applications. You can also use other tools such as Robusta, Pixie, New-relic, and other monitoring tools with Devtron, as it provides the flexibility to integrate with any tool of your choice.

Feel free to check out the documentation for more info, and if you like Devtron, do give it a star on GitHub.